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FOREWORD 


The  Thirty-Fourth  Conference  on  the  Design  of  Experiments  in  Army  Research, 
Development  and  Testing  was  held  on  19-21  October  1988  in  the  auditorium  of  the 
Physical  Sciences  Laboratory  on  the  campus  of  New  Mexico  State  University,  Las 
Cruces,  New  Mexico.  Mr.  John  Lockert,  Director  of  the  White  Sands  Missile 
Range,  stated  his  Installation  would  serve  as  the  host  for  this  meeting.  He 
selected  Mr.  William  Agee  to  act  as  the  chairperson  for  local  arrangements. 

The  attendees  appreciated  the  quiet  and  efficient  manner  In  which  this 
gentleman  handled  the  many  tasks  associated  with  this  event.  He  Is  also  to  be 
commended  for  his  planning  arrangements  for  a  tutorial  which  was  scheduled  to 
be  held  two  days  beforo  the  start  of  this  conference. 

The  original  format  for  the  Oesign  of  Experiments  Conferences,  which  are  under 
the  auspices  of  the  Army  Mathematics  Steering  Committee  (AMSC),  was  outlined  by 
the  eminent  statistician.  Professor  Samuel  S.  Wilks,  who  served  as  conference 
chairman  until  his  death.  Through  these  symposia  the  AMSC  hopes  to  Introduce 
and  encourage  the  use  of  the  latest  statistical  and  design  techniques  into  the 
research,  development  and  testing  conducted  by  the  Army's  scientific  and 
engineering  personnel.  It  Is  believed  that  this  purpose  can  be  best  pursued  by 
holding  these  meetings  at  various  government  installations  throughout  the 
country. 

Members  of  the  program  committee  were  pleased  to  obtain  the  services  of  the 
following  distinguished  scientists  to  speak  on  topics  of  Interest  to  Army 
personnel : 


Speaker  and  Affiliation 

Professor  Herbert  A,  David 
Iowa  State  University 

Professor  Ronald  R.  Hocking 
Texas  A&M  University 

Professors  Donald  L.  Iglehart 
and  Peter  W.  Glynn 
Stanford  University 


Title  of  Address 

Some  Applications  of  Order 
Statistics 

Diagnostic  Methods  -  Variance 
Component  Estimation 

Computational  and  Statistical 
Issues  in  Discrete-Event 
Si  mul  ati  on 


Professor  Emanuel  Parzen 
Texas  A&M  Uni versl ty 

Professor  Edward  L.  Wegman 
George  Mason  University 


Two  Sample  Functional 
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Parallel  Coordinate 
Density  Plots 
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Four  days  before  the  start  of  the  planned  two-day  tutorial  on  "Topics  in  Modern 
Regression  Analysis",  its  speaker  advised  Mr.  Agee  he  could  not  give  his 
planned  lectures.  Fortunately,  Professor  A1 i  Hadi  of  Cornell  University  was 
able,  so  to  speak,  to  save  the  day.  The  attendees  were  very  pleased  with  Or. 
Hadi ' s  interesting  and  informative  tutorial  on  "Sensitivity  Analysis  in  Linear 
Regression". 

Dr.  Marion  R.  Bryson,  Director  of  the  U.S.  Army  Combat  Development 
Experimentation  Center,  was  the  recipient  of  the  eighth  Wilks  Award  for 
Contributions  to  Statistical  Methodologies  In  Arny  Research,  Development  and 
Testing.  This  honor  was  bestowed  on  Dr.  Bryson  for  his  many  significant 
contributions  to  the  field  of  statistics.  These  started  by  providing 
statistical  consulting  wnile  he  was  on  the  faculty  of  Duke  University.  This 
era  was  followed  by  full-time  work  devoted  to  directing  analytical  studies  for 
the  Army.  Since  then,  he  has  provided  overall  technical  direction  to  the 
Army's  most  modern  field  test  facility.  His  published  works  include  papers  on 
a  wide  range  of  topics  of  importance  to  the  Army,  including  methods  for  scoring 
casualties,  designing  field  experiments,  and  inventory  control  problems. 

The  AMSC  has  asked  that  these  proceedings  be  distributed  Army-wide  to  enable 
those  who  could  not  attend  this  conference,  as  well  as  those  that  were  present, 
to  profit  from  some  of  the  scientific  ideas  presented  by  the  speakers.  The 
members  of  the  AMSC  are  taking  this  opportunity  to  thank  all  the  speakers  for 
their  interesting  presentations  and  also  members  of  the  program  committee  for 
their  many  contributions  to  this  scientific  event. 
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SOME  APPLICATIONS  OF  ORDER  STATISTICS* 
H.  A.  David 

Department  of  Statistics 
102D  Snedecor  Kali 
Iowa  Stats  University 
Ames,  IA  500 11-1210 


ABSTRACT.  Suppose  that  the  random  variables  . xn  are  arranged  in 

ascending  order  as  X,  <  ...  <  X.  .  Then  X,.,n  is  called  the  r-th  order 

l  sn  n in  r  .n 

statistic  (r  -  i,...,n).  Order  statistics,  and  functions  thereof,  have  been 
used  extensively  in  such  diverse  areas  as  quality  control,  the  estimation  of 
parameters,  life  testing,  data  compression,  selection  procedures,  and  the 
study  of  extreme  meteorological  phenomena.  In  this  paper  we  focus  on 
applications  of  order  statistics  to  (a)  estimators  that  are  resiutant  to 
outliers,  (b)  current  measures  of  location  and  dispersion  such  as  the  moving 
median  and  the  moving  range,  and  (c)  some  problems  in  reliability. 


1.  INTRODUCTION.  If  the  random  variables  Xp...,Xn  are  arranged  in 
ascending  order  of  magnitude  and  then  written  as 


X,  < 
l:n 


...  <  xr;n  <  ...  <  xn{n, 


we  call  Xr;n  the  r-th  order  statistic  (0S)(r  *  l,...,n).  Usually  X,,...,Xn 
are  assumed  to  be  a  random  sample  from  some  underlying  population. 


The  subject  of  order  statistics  deals  with  the  properties  and  applica¬ 
tions  of  these  ordered  random  variables  and  of  functions  Involving  them. 
Examples  are  the  extremes  X|  >n  and  Xnjn,  the  range  V?n  -  Xn.n  -  Xlsn,  the 

extreme  deviate  (from  the  sample  mean)  X  -  X,  and  the  maximum  absolute 

11  ■  n:n  ' 

deviation  from  the  median  (MAD)  ._,max  lx.  -  Ml .  where  the  median  M 
1  .  , . . . ,n 1  l  1 

equals  X  (n  odd)  and  ^(X  +  X  )  (n  even). 

(SjV  2  (§)’"  ♦  1)'" 

All  these  statistics  have  Important  application.  The  extremes  arise  in 
the  statistical  study  of  droughts  and  floods,  as  well  as  in  problems  of 
breaking  strength  and  fatigue  failure.  The  range  is  well  known  to  provide  a 
quick  estimator  of  the  population  standard  deviation  o,  whereas  MAD  is  a  more 
recent  estimator  of  o  valuable  because  of  its  high  resistance  to  wild 
observations  (outliers).  The  extreme  deviate  is  a  basic  tool  in  the  detection 
of  such  outliers,  large  values  of  (X  -  X)/o  indicating  the  presence  of  an 
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standardized  bias  and  mean  squared  error  (MSF.)  of  L  (a)  under  (2.1)  can  be 

n 

obtained  with  the  help  of  tables  of  the  first  two  momenta  of  normal  order 

statistics  in  the  presence  of  an  outlier  (David,  Kennedy,  and  Knight,  1977), 

For  example,  under  (2.1)  the  standardized  bias  E  (X)  of  L  (a)  is  given  by 

n  n 


E  L  (a)  -  y 
n 


r  „  r^lsn  1 

2  atE  [ - 5 - ), 


or 


b(X)  -  2  a. a  (X),  (2.3) 

n  i  l  *  n 

where  aJ  (X)  is  the  expected  value  of  X4 for  y  ■  0,  o  ■  1.  Note 

1 !  n  i .  n  , 

that  a.  (0)  is  just  the  widely  tabulated  expected  value  a  of  the  i-th  OS, 

zi-n*  in  ran^ora  samples  of  n  from  a  standard  normal  population.  Clearly, 

a  (X)  is  a  strictly  increasing  function  of  X,  Also,  since  for  X  ■  »,  (2.1) 

i  •  n 

leaves  us  with  a  random  normal  sample  of  size  n  -  1  plus  an  observation  at 

i  -  1 . .  anjn  (-)  -  •,  (2.4) 


we  have 


ot.  (®) 
i:n 


itn-l* 


(and  likewise  a,  (-*•) 
isn 


i  ■  2 , . . .  ,  n 


i-1 :n' 

Some  results  for  samples  of  10  are  shown  in  Figures  1  and  2,  where 

!  10 

X1Q  is  the  sample  mean 2  X^q 

,  9 

T  1Q( 1 )  is  the  trimmed  mean  g  2  Xi*10 

l  8 

W1Q(2)  is  the  Winsorized  mean  C  2X3 . 1Q  +  2  Xi<10  +  2X8 . 10 ) 


Tjq(4)  is  the  median  ?K,in  +  *Slln).  etc. 


2^  5:10 


6:10' 


The  figures  are  confined  to  X  >  0  since  results  for  X  <  0  follow  by  skew- 
symmetry  in  Fig.  1  and  by  symmetry  in  Fig.  2. 

BIAS  bn(X). 


Since  a^  >  0 V^i  we  see  from  (2,3)  that  the  bias  is  a  strictly 


Increasing  function  of  X  for  each  of  the  estimators,  and  from  (2.4)  that 

(2.5) 


v-> 


1  *1 


This  gives  the  numerical  values  placed  on  the  right  of  Fig.  1.  The  jagged 
graphs  are  the  corresponding  "stylized  sensitivity  curves"  (Tukey,  1970; 

Andrews  et  al.,  1972)  obtained  by  plotting  L  (a,  a  ,  , ,X) 

n  i:n— l  n-l:n-l 

against  X.  In  particular,  for  the  median  we  have 


3 


tned  ^  q 


^a5:9  + 

2^a5:9  +  “6:9^ 


2  a6:9 


0  <  X  <  o, 


6:9 


X  >  a 


6:9 


0.1372  6 


The  last  result  Is  the  same  as  given  by  (2.5).  In  fact,  each  of  the  horizon¬ 
tal  lines  serves  as  an  asymptote  to  the  corresponding  bias  function.  It  is 
seen  that  the  median  performs  uniformly  best. 

MEAN  SQUARED  ERROR  MSE(X)  ♦  No  clear-cut  results  emerge.  The  sample  mean 
does  best  for  X  <  1.5  but  is  quickly  outclassed  for  larger  X,  Overall,  Tjq(1) 

performs  best,  although  the  more  highly  trimmed  T^q(2)  is  slightly  superior 

for  very  large  X. 

EXTENSIONS 


1.  The  intuitively  appealing  result  that  for  symmetric  unimodal  distributions 
the  median  is  the  least  biased  among  the  Ln-estimators  can  be  formally 

established  under  (2.1)  and  also  for  a  class  of  symmetric  non-normal 
distributions  (David  and  Ghosh,  1985). 


2.  For  n  <  20  appropriately  trimmed  means  still  do  well  in  the  MSE  sense  when 
compared  with  much  more  complex  estimators,  but  fop  X  sufficiently  large 
and  n  not  too  small  are  inferior  to  the  best  of  the  adaptive  estimators 
such  as  Tukey's  biweight  (Mosteller  and  Tukey,  1977,  p.  205). 


3.  An  often  used  alternative  outlier  model  replaces  the  second  line  of  (2.1)  by 

X1  ~  N(w,  tV)  T2  >  1 

For  this  model  location  estimators  remain  unbiased  but  their  variance  is 
increased.  Since  bias  has  been  sidestepped,  only  the  variance  of  the 
estimator  needs  to  be  studied  (David  and  Shu,  1978;  Roaenberger  and  Gasko, 
1983). 

CASE  OF  SEVERAL  EXTREME  OUTLIERS.  For  q(l  <  q  <  |  n)  outliers  Rocke  (1986) 

defines  as  a  measure  of  outlier  resistance  of  an  estimator  of  location  T  the 
"expected  maximum  bias"  D^(n,q)  by 


DT(n,q)  -  E{sup[ |TCz i :n-q» 


,Z  ,J 

*  n-qin-q* 


(2.6) 


where  the  supremum  is  taken  over  all  possible  choices  of  the  constants 
X j , ,Xq  and  the  Z's  are  the  normal  OS.  When  T  ■  Ln,  the  supremum  will 

evidently  occur  when  the  X’s  are  all  +  *  or  all  -  •».  As  Rocke  points  out,  by 
focusing  on  the  worst  case  of  bias  one  need  not  specify  the  usually  unknown 
diatributlon(s)  of  the  outliers.  It  suffices  to  model  the  good  observations 
which  more  generally  could  be  from  any  standardized  distribution. 


It  appears  that  unwittingly  Rocke  does  not  use  (2,6)  but  in  fact  works 
with  the  standardized  bias 
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EtTtzi!n_q 


,2  »*.•••*•)}■ 
’  n-q : n-q  *  *  ’  ' ‘ 


(2.7) 


If  the  good  observations  were  independently  generated  from  a  unlmodal 
symmetric  distribution  (mode  ■  maximum),  then  again  the  median  can  be  shown  to 

have  the  least  bias  D^(n,q)  among  ^-statistics  (Rocke's  proof  is  incorrect; 

see  the  appendix). 


3.  CURRENT  MEASURES  OF  LOCATION  AND  DISPERSION 

Let  X.  be  a  sequence  of  Independent  random  variables  with  cdf  F.  (x) 

(i  ■  1,2,...).  Then  '  «  (Xj , . . .  *x1+.n_j )  may  b®  called  a  moving  sample  of 

size  n.  and  X^  .  the  r-th  OS  of  S  the  moving  r-th  OS.  Moving  maxima 

t  *  n  n  *  ■  1  - 

(r  ■  n)  and  minima  (r  ■  1)  were  studied  by  David  (1955)  under  homogeneity 

(F^(x)  ■  F(x),  i  ■  1,2,...,)  in  the  course  of  an  investigation  of  moving 

ranges  W  ^  -  X^^  (i  ■  1.2....).  The  latter  have  a  longer  history 

ti  ti  •  n  l  in 

(Grant,  1946),  being  natural  companions  to  moving  averages  on  quality  control 
charts.  Such  charts  are  particularly  appropriate  when  it  takes  some  time  to 
produce  a  single  observation. 

Moving  medians  are  robust  current  measures  of  location  and,  like  moving 
averages,  smooth  the  data;  see,  e.g.,  Tukey  (1977,  p.  210).  Cleveland  and 
Kleiner  (1975)  have  used  the  moving  midaean.  the  mean  of  the  central  half  of 

the  ordered  observations  in  each  Sfl^,  together  with  the  mean  of  the  top  half 

and  the  mean  of  the  bottom  half,  as  three  moving  descriptive  statistics 
indicating  both  location  and  dispersion  changes  in  a  time  series. 

Since  S_^  and  S„^  involve  common  random  variables  iff 

d  =  li  -  j|  <  n,  we  see  that  X'1'  and  X'^'  are  independent  for  d>n  and 
1  1  r  $  n  b  •  n 

dependent  otherwise,  with  n  -  d  rv’s  in  common.  To  begin  with,  we  assume 

homogeneity.  Then  the  Joint  distribution  of  Sn^  and  Sn^  ^  will  be 

stationary  and  will  depend  only  on  F(x),  n,  and  d.  We  therefore  consider 

and  S„^+d\  and  more  specifically  X^  and  X^*+d^ 
n  n  rsn  ssn 

(r ,s  ■  l, . . . ,n).  Let 

Ttgh<d)  “  Pr{rank  (Xp)n)  "  8.  rank  "  M*  (3.1) 


where  rank  (Y)  denotes  the  rank  of  Y  in  the  combined  sample  Xj , . . . ,Xn+d .  It 
follows  that 
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(3.2) 


YhV  E(Vn*l  ,ih!.+d)- 


This  permits  calculation  of  cov(x^ ,  X^+<^ )  in  terms  of  the  first  two 
moments  of  order  statistics  in  samples  of  n+d  from  a  distribution  with  cdf 
F(x),  since  the  *  .  can  be  obtained  by  combinatorial  arguments  (David  and 

8"  ( J  \  / 4  \ 

Rogers.  1983).  The  joint  distribution  of  X'  '  and  XVJ/  has  been  investigated 
by  Inagaki  (1980). 

With  the  help  of  (3.2)  it  is  possible  to  evaluate  the  auto-covariance 

structure  under  homogeneity  of  the  moving  median  and.  in  fact,  of  any  linear 

function  of  the  order  statistics  .  -  «,x,  +  ...  +  a  x  .  That  is,  we 

l  i:n  nn:n 

can  find 

cov 


in  terras  of  the  first  two  moments  of  the  OS  for  sample  sizes  up  to  2n-l  from  a 
distribution  with  cdf  F(x). 

Electrical  engineers  have  made  extensive  use  of  moving  order  statistics 
in  digital  filters.  They  view  a  moving  sample  as  a  window  on  a  sequence  of 
signals  Xj,  x2, ...  and  speak  of  median  filters  when  using  the  moving  median  to 

represent  the  current  value  of  the  signal,  thereby  "filtering  out"  occasional 
Impulsive  disturbances  (outliers)fu.g. ,  Arce,  Gallagher,  and  Nodes,  1986). 

More  generally,  the  median  may  be  replaced  by  «'x  to  give  order  statistic 
filters  (e.g.,  Bovlk  and  Restrepo,  1987). 

For  example,  suppose  that  in  the  automatic  smoothing  of  a  basically 
stationary  time  series  one  is  prepared  to  Ignore  single  outliers  but  wishes  to 
be  alerted  to  a  succession  of  two  or  more  high  (or  low)  values.  This  calls 
for  use  of  moving  medians  in  samples  of  three,  since  clearly  a  single  outlier 
will  be  smoothed  out  but  two  successive  large  values  will  result  in  two  large 


medians. 

purposes 

The  following 
of  comparison 

small  example 
we  have  added 

illustrates 
the  much  less 

the  situation 
informative 

,  where 
moving 

i  for  _ 
mean  x^. 

*i 
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1 

10 
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3 

9 

10 

2  1 

x(1) 

*2:3 

1 
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4 

9 

9 

2 
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4  4 

5-  3 

33  J 

53 

7 

When  XpX2»«..  are  not  iid,  even  the  distribution  of  order  statistics  in 
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a  fixed  sample  becomes  complicated  although  a  fairly  elegant  expression  for 
the  pdf  can  be  written  down  in  terms  of  permanents  (Vaughan  and  Venables, 

1972)  if  the  X's  are  not  Identically  distributed  but  still  Independent,  It  is 
easily  seen  that  the  moving  median  and  other  order  statistics  will  reflect 
trends  except  for  a  smoothing  at  the  ends.  Thus  for  the  following  sequence, 
where  the  upward  trend  is  underlined,  we  have 

9  12  14  11  7 


9  12  12  11 

9  1 1|  12-|  lo| 


(3.3) 


with  covariances  cov  (x^ ,  X^)  (r,s  ■  l,...,n)  independent  of  i. 

r  •  n  s  *  n 

Consider  now  a  particular  sample  X! ,X2, .. (o  -  2,3,...)  with 

symmetric  unimodal  distributions.  Then  under  (3.3),  which  need  hold  only  for 
the  sample  in  question,  we  see  that  for  T>0 

Prjrank  Xi  •  i}  increases  with  t. 

Thus  Xr;n  will  tend  to  lead  the  trend,  reflect  the  current  state,  or  lag  the 

trend  according  as  r  ^  m,  and  will  do  so  increasingly  as  t  Increases;  for 

T<0,  the  results  are  reversed.  However,  in  contrast  to  the  sample  mean, 
whose  variance  remains  unchanged  under  a  linear  trend,  the  variance  of  the 
sample  median  increases  with  t.  (I  am  indebted  to  Dr.  W.  J.  Kennedy  for  some 
computations  verifying  the  latter  intuitively  obvious  result.)  Thus  the  use 
of  the  median,  under  locally  linear  trend,  is  appropriate  primarily  as 
protection  against  outliers.  In  this  situation,  but  under  nonlinear  trend, 
Bovik  and  Naaman  (1986)  consider  the  optimal  estimation  of  EXffl  by  linear 
functions  of  order  statistics. 

4.  SOME  PROBLEMS  IN  RELIABILITY 


xi  5  2  1  3  4  6 


For  a  linear  trend  given  by 

Xt  -  ix  +  Zt,  i  -  1,2,... 

where  the  Z^  are  l.i.d.,  we  evidently  have 

“r:n<T>  -  “rm<T)  '  ‘J-1*’- 


There  is  a  well-known  immediate  connection  between  order  statistics  and 
the  reliability  of  k-out-of-n  systems. 

Definition  A  k-out-of-n  system  is  a  system  of  n  components  that  functions  if 
and  only  if  at  least  k  (k  ■  l,...,n)  of  its  components  function.  Series  and 
parallel  systems  correspond  to  k  ■  n  and  k  ■  1. 
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LeC  Xf(i  -  l,...,n)  be  Che  lifetime  of  the  i-th  component  and 
R^x)  -  Pr^  >  x}  its  reliability  at  time  x  (the  probability  that  it  will 
function  at  time  x).  Then  the  reliability  of  the  system  S  at  time  x  is 
y*>  -  Pr{Xn_k+lltl  >  «|. 

If  the  Xj  are  independent  (but  not  necessarily  identically  distributed)  one 
may  write  (Sen,  1970;  Pledger  and  Proschan,  1971), 

n  A.  1-A, 

Rg(x)  -in  V(x)  [1  -  Ri(x)]  \ 

A  i-1 

n 

where  A.  ■  0  or  1,  and  A  is  the  region  £  A.  >  k.  It  ca  be  shown  that  a 
1  i-1  1 

series  (parallel)  system  is  at  most  (least)  as  reliable  as  the  corresponding 

1  n 

system  of  components  each  having  reliability  R(x)  -  -  l  R.  (x).  An  excellent 

n  i-1  1 

general  account,  covering  also  important  situations  when  the  X^  are  not 
independent,  is  given  in  Barlow  and  Pros chan,  (1975). 

I  will  conclude  with  a  problem  In  reliability,  quite  different  from  the 
above,  that  was  suggested  by  an  enquiry  from  Malcolm  Taylor  (see  Baker  and 
Taylor,  1981).  A  fuze  contains  n  detonators,  r  of  which  must  function  within 
time  span  t.  The  ideal  requirement  r  >  n  may  be  too  demanding  in  practice  and 
r  ■  n-1  suffices.  The  n  times  to  detonation,  Xp...,Xn,  may  reasonably  be 

regarded,  I  was  told,  as  a  random  sample  from  a  normal  population.  Let 

P(r;  n,t)  be  the  probability  that  at  least  r  detonations  have  occurred  in  time  t. 

Now,  for  a  random  sample  from  any  continuous  distribution  with  cdf  F(x), 
P(n;n,t)  is  just 

PtiXn!n  “  Xl:n  <  M  "  n/IjF<*+t>  “  F<*>Jn~l  «(*>. 
the  cdf  of  the  sample  range  (Hartley,  1942).  Let  A}'  and  A2  be  the  events 

Xn-1 :n  "  Xl:n  «  *  and  Xnsn  "  X2Sn  <  *•  Then* 

P(n-1;  n,t)  -  PrjAj'U  A2} 

-  Pr { Aj * }  +  Pr (Aj }  -  PrfAj'Aj}  (4.1) 

The  event  A^’  occurs  if  n-1  or  n-2  of  Xp...^  fall  in  the  interval 

(Xi in*xl jn  +  and  A2  n“2  of  the  Xi  ara  tn  ^X2sn,X2:n  +  since  A2 

includes  the  event  that  n-1  of  the  X^  are  in  (Xi5n,Xj.n  +  t],  we  can  avoid 

unnecessary  duplication  by  replacing  A^'  in  (4,1)  by  Ap  the  event  that 

exactly  n-2  of  the  X,  are  in  (X,  ,  X,  +  tl  . 

1  1  in  1 :n  J 
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We  have  immediately,  writing  -  n(n-l)  ...  (n-j  +  1),  that 

Pr{ Aj }  -  n^j".  [F(x+t)  -  F<x)]tV"2[l  -  F(x+t)]  dF(x) 
and 

Pr{A2}  ■  n^/^  [F(x+t)  -  F(x)]n  2  F(x)dF(x). 

The  joint  occurrence  of  Aj  and  Aj  is  llluitrated  below  for  n  ■  6.  We 

have 


x— 
Xl:n  ■  x 


i  ■ 

x+t 


PrUjAj}  -  n(3)/l./*+ttF(x+t)  -  F(y) ] n“3[F(y+t)  -  F(x+t)]  dF(y)dF(x) 


From  these  results  P(n-lj  n,t)  has  been  tabulated  in  David  and  Kinyon 

(1983)  when  F(x)  ■  $(x).  Note  that  P(n-l;n,t)  may  be  interpreted  as  the 

o 

probability  that  at  least  n-1  out  of  n  Independent  normal  N(y,o  )  variates  are 
within  an  Interval  of  length  to. 

EXAMPLE.  As  in  Baker  and  Taylor  (1981)  suppose  that  X,,...tXn  are  independent 

normal  variates  with  o  »  10~5.  The  entry  P(6;7,3)  »  0.9587  tells  us  that  the 
probability  of  at  least  six  detonations  out  of  a  possible  seven  within  time 
span  3o  la  0.9587.  By  comparison,  the  probability  of  seven  detonations  is 
only  0.6601,  as  found  from  tables  of  the  cdf  of  the  range  (Pearson  and 
Hartley,  1970). 


David  and  Kinyon  (1983)  also  give  an  expression,  involving  a  triple 
Integral,  for  P(n-2;  n,t).  It  should  be  noted  that  P(r;  n,t)  han  received 
much  attention  by  quite  different  techniques  in  the  special  case  when  the  X^ 
are  Independent  uniform  variates  (e.g.,  Neff  and  Naus,  1980).  From  a 
different  viewpoint  again,  writing 


where 


P(r;  n,t)  -  Pr{Hn(J)  <  t}, 


we  may  regard  Hn(jjj)  as  a  measure  of  dispersion.  Zn  fact,  HR(n)  is  the  length 
of  the  shorth,  the  shortest  a-fraction  of  the  ordered  sample  (Andrews  et  al., 
1972).  It  has  recently  been  shown  (Grubel,  1988)  that  Hn(a)  Is  asymptotically 
normal  (for  fixed  ot). 
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APPENDIX 

H.  A.  David  and  C.  C.  Yang 
Correction  to  'Outlier  resistance  in  small  samples' 

By  DAVID  M.  ROCKE 
Biometrika  (1986),  73,  175-81 

The  author  does  not  stay  with  hia  own  definition  of  D<f(n,q)  but  in  fact  uses 

DT(n,q)  -  E{T(*l,...,an_q>«, 

Even  with  this  change  the  proof  of  the  theorem  on  p,  176  is  in  error  since  the 

combinatorial  term  associated  with  5  should  be  not  fn  q).  However, 

n-r  vn-r;  K  t  '  1 

since  6  -  6  ,  the  theorem  follows  directly  from  Case  2  of  David  and 

n-r  r-q '  * 

Groeneveld  (Biometrika  (1982),  69,  227-32)  and  has  essentially  been  proved  in 
P.  K.  Sen  (Ed.)  Blostatlstics  (1985)  North-Holland,  pp.  309-11. 
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MULTI-SAMPLE  FUNCTIONAL  STATISTICAL  DATA  ANALYSIS 


Emanuel  P&rzen 
Department  of  Statistics 
Texas  A&M  University 
College  Station,  Texas  77843-3143 


ABSTRACT.  This  paper  discusses  a  functional  approach  to  the  problem  of  compar¬ 
ison  of  multi-samples  (two  samples  or  e  samples,  where  e  >  2).  The  data  consists  of  e 
random  samples  whose  probability  distributions  are  to  be  tested  for  equality.  A  diversity 
of  statistics  to  test  equality  of  c  samples  are  presented  in  a  unified  framework  with  the 
aim  of  helping  the  researcher  choose  the  optimal  procedures  which  provide  greatest  insight 
about  how  the  samples  differ  in  their  distributions.  Concepts  discussed  are:  sample  distri¬ 
bution  functions;  ranks;  mid-distribution  function;  two-  sample  t  test  and  nonparametric 
Wilcoxon  test;  multi-sample  analysis  of  variance  and  Kruskai  Wallis  test;  Anderson  Darling 
and  Cramer  von  Mises  tests;  components  and  linear  rank  statistics;  comparison  distribu¬ 
tion  and  comparison  density  functions,  especially  for  discrete  distributions;  components 
with  orthogonal  polynomial  score  functions;  chi-square  tests  and  their  components. 

1.  INTRODUCTION.  We  assume  that  we  are  observing  a  variable  Y  in  c  cases  or  sam¬ 
ples  (corresponding  to  e  treatments  or  c  populations).  The  samples  can  be  regarded  as  the 
value  of  e  variables  Y\f . . . ,  Yc  with  respective  true  distribution  functions  Fi(y), . . . ,  F?(y) 
and  quantile  functions  CJi(u), . . . ,  Qc( u).  We  call  Yi, . . . ,  Yc  the  conditioned  variables  (the 
value  of  Y  in  different  populations). 

The  general  problem  of  comparison  of  conditioned  random  variables  is  to  model  how 
their  distribution  functions  vary  with  the  value  of  the  conditioning  variable  k  =» 
and  in  particular  to  test  the  hypothesis  of  homogeneity  of  distributions: 

J?0  :  Fj  *...*■  Fe  =■  F 

The  distribution  F  to  which  all  the  others  are  equal  is  considered  to  be  the  unconditional 
distribution  of  Y  (which  is  estimated  by  the  sample  distribution  of  Y  in  the  pooled  sample). 

8.  DATA.  The  data  consists  of  e  random  samples 

Yls(j)» 3  **  li  •  •  •  * nh 

for  k  —  1, . . . ,  c.  The  pooled  sample,  of  size  N  **  n\  +  . , .  +  ne,  represents  observations  of 
the  pooled  (or  unconditional)  variable  Y .  The  c  samples  are  assumed  to  be  independent 
of  each  other. 

8.  SAMPLE  DISTRIBUTION  FUNCTIONS.  The  sample  distribution  functions  of 
the  samples  are  defined  (for  -oo  <  y  <  oo)  by 

Ffe"(y)  =  fraction  <  y  among  ¥*(.). 

The  unconditional  or  pooled  sample  distribution  of  Y  is  denoted 

F*(y)  -  fraction  <  y  among  Yfe(.)>  fc  —  1, ...» e. 

We  use  *  to  denote  a  smoother  distribution  to  which  we  are  comparing  a  more  raw 
distribution  which  is  denoted  by  a  An  expectation  (mean)  computed  from  a  sample  is 
denoted  JET. 

Research  Supported  by  the  U.S.  Army  Research  Office 
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4.  RANKS,  MID-RANKS,  AND  MID-DISTRIBUTION  FUNCTION.  Nonparamet- 
ric  statistics  use  ranks  of  the  observations  in  the  pooled  sample;  let 

Rle(t)  denote  the  rank  in  the  pooled  sample  of  !%(*). 

One  can  define  Rk{t)  =  NF'^Y^t)). 

In  defining  linear  rank  statistics  one  transforms  the  rank  to  a  number  in  the  open  unit 
interval,  usually  J R^(t)/(iV  4- 1).  We  recommend  {Rie[t)  -  .5 )/N.  These  concepts  assume 
all  observations  are  distinct,  and  treat  ties  by  using  average  ranks.  We  recommend  an 

approach  which  we  call  the  “mid-rank  transform”  which  transforms  Y^t)  to  !%(*)), 
defining  the  mid-distribution  function  of  the  pooled  sample  Y  by 

P*[y)  =  r(y)  -  .5p*(y). 

We  call 

PA(y)  ~  fraction  equal  to  y  among  pooled  sample 
the  pooled  sample  probability  mass  function. 

5.  SAMPLE  MEANS  AND  VARIANCES.  When  the  random  variables  are  assumed 
to  be  normal  the  test  statistics  are  based  on  the  sample  means  (for  k  =  1, . . . ,  e) 

n--*nni-(i/nfc)  £>*(<)• 

t»l 

We  interpret  Y)f  as  the  sample  conditional  mean  of  Y  given  that  it  comes  from  the  fcth 
population.  The  unconditional  sample  mean  of  Y  is 

Y-  =  fT|y)  =  p.iiv  +  . . .  +  P.,YC-, 

defining 

P.Jfe  =  *k/N 

to  be  the  fraction  of  the  pooled  sample  in  the  fcth  sample;  we  interpret  it  as  the  empirical 
probability  that  an  observation  comes  from  the  fcth  sample. 

The  unconditional  and  conditional  variances  are  denoted 

VAR-[y|  =  (1  IN)  22  22{Yk(j)  -  y-}* 

fowl  jml 

ni , 

vARini  =  a/-**)  £{n  O')  -  n-}2 

j'=l 

Note  that  our  divisor  is  the  sample  size  N  or  rtjt  rather  than  JV  -  c  or  -  1.  The  latter 
then  arise  as  factors  used  to  define  F  statistics. 

We  define  the  pooled  variance  to  be  the  mean  conditional  variance: 


*‘2  =  I>.t  VAR-[nj 

k-l 
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6.  TWO  SAMPLE  NORMAL  T  TEST .  In  the  two  sample  case  the  statistic  to  test 
Hq  is  usually  stated  in  a  form  equivalent  to 

r  =  {n-  -  -  2))((i/m)  +  (i/»2))}-8 

We  believe  that  one  obtains  maximum  insight  (and  analogies  and  extensions)  by  expressing 
T  in  the  form  which  compares  Yi  "  with  Y~\ 

T  =  {(JV  -  2)p.i/(l  -  p.i)}-5{n-  -  Y-)l°‘ 

The  exact  distribution  of  T  is  t(N  -  2),  t-distribution  with  N  -  2  degrees  of  freedom. 

7.  TWO-SAMPLE  NONPARAMETRIC  WILCOXON  TEST .  To  define  the  popular 
Wilcoxon  non-parametric  statistic  to  test  Hq  wo  define  Wk  to  be  the  sum  of  the  nk  ranks 
of  the  Yk  values;  its  mean  and  variance  are  given  by 

E[Wk\  =  nk(N  +  l)/2,  VARfWjfe]  =  +  1)/12 

The  usual  definition  of  the  Wilcoxon  test  statistic  is 

Tk  -  {Wk  -  *[^l}/{VAR[IVt]}-5. 

The  approach  we  describe  in  this  paper  yields  as  the  definition  of  the  nonparametric 
Wilcoxon  test  statistic  (which  can  be  verified  to  approximately  equal  the  above  definition 

of  Tt,  up  to  a  factor  {1  -  (l/AT)2}’5) 

Tx  »  {12(JV  -  l)p.i/(l  -  P.l)}-!(*f  -  .5), 

defining 

•*)/* 

=  M/nJn  -  (1/2JV-) 

One  reason  we  prefer  this  form  of  expressing  non-parametric  statistics  is  because  of  its 
relation  to  mid-ranks; 

st-=fr[r-(n)l 

One  should  notice  the  analogy  between  our  expressions  for  the  parametric  test  statistic 
T  and  the  nonparametric  test  statistic  T\\  the  former  has  an  exact  t(N  -  2)  distribution 
and  the  latter  has  asymptotic  distribution  Normal{0,  l}. 

8.  TEST  OF  EQUALITY  OF  c  SAMPLES  NORMAL  CASE.  The  homogeneity  of 
c  samples  is  tested  in  the  parametric  normal  care  by  the  analysis  of  variance  which  starts 
with  a  fundamental  identity  which  in  our  notation  is  written 

VAR'|i'|  =  £  P.k{Yk~  ~  J"}2  +  <r'! 

Jfc=l 

The  F  test  of  the  one-way  analysis  of  variance  can  be  expressed  as  the  statistic 

r2  =  £Pjt|rfc|2, 

1 

k=  1 
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defining 


n  =  (jv-«){y*--y-}/«- 

TFt  =  {(JV  -  c)p.k/(l  -  p.k)}  s{Yk-  -  y-}/<7* 

The  asymptotic  distribution  of  T2/(c  - 1)  and  TF%  are  F(c  - 1, 7/  -  e)  and  F(l,  N  -  e) 
respectively. 

9.  TEST  OF  EQUALITY  OF  e  SAMPLES  NONPARAMETRIC  KRUSKAL- 
WALLIS  TEST.  The  Kruskal- Wallis  nonpaxametric  test  of  homogeneity  of  c  samples 
can  be  shown  to  be 


TKW*  =  £(1  -  p.k)\TKWk\*. 

fcssl 

TKWk  =  {U(JV  -  l)p.*/(l  -  p.k))*(Rk  -  .5} 

The  asymptotic  distributions  of  TKW 2  and  TKW%  are  chi-squared  with  c  -  1  and  1 
degrees  of  freedom  respectively. 

10.  COMPONENTS.  We  have  represented  the  analysis  of  variance  test  statistic  T2 
and  the  Kruskal- Wallis  test  statistic  TKW 2  as  weighted  sums  of  squares  of  statistics  TF^ 
and  TKWig  respectively  which  we  call  components,  since  their  values  should  be  explicitly 
calculateu  to  indicate  the  source  of  the  significance  (if  any)  of  the  overall  statistics.  Other 
test  statistics  that  can  be  defined  can  be  shown  to  correspond  to  other  definitions  of 
components. 

11.  ANDERSON  DARLING  AND  CRAMER  VON MISES  TEST  STATISTICS.  Im¬ 
portant  among  the  many  test  statistics  which  have  been  defined  to  test  the  equality  of 
distributions  are  the  Anderson-Darling  and  Cramer-von  Mises  test  statistics.  They  will 
be  introduced  below  in  terms  of  representations  as  weighted  sums  of  squares  of  suitable 
components. 

12.  COMPARISON  DISTRIBUTION  FUNCTIONS  AND  COMPARISON  DEN- 
SutY  FUNCTIONS.  We  now  introduce  the  key  concepts  which  enable  us  to  unify  and 
choose  between  the  diverse  statistics  available  for  comparing  several  samples.  To  compare 
two  continuous  distributions  F(.)  and  H[.)t  where  H  is  a  true  or  smooth  and  F  is  a  model 
or  raw,  we  define  the  comparison  distribution  function 

D(u)  =  £>(«;//,  F)  =  F(H~l{u)) 


with  comparison  density 

d(u)  =  d(u',H,F)  =  D'{  u)  =  /(H_1(u))//i(ir~1(tt)). 

Under  Hq  :  H  =  F,  D( u)  =  u  and  d(u)  =  1.  Thus  testing  Hq  is  equivalent  to  testing 
D(u)  for  uniformity. 

Sample  distribution  functions  are  discrete.  The  most  novel  part  of  this  paper  is  that 
we  propose  to  form  an  estimator  from  estimators  H~ jf.l  and  F*(.)  by  using  a  general 
definition  of  D[.)  for  two  discrete  distributions  H(.)  and  F(.)  with  respective  probability 
mass  functions  p  w  and  pp  satisfying  the  condition  that  the  values  at  which  pji  are  positive 
include  all  the  values  at  which  pp  are  positive. 


IS.  COMPARISON  OF  DISCRETE  DISTRIBUTIONS.  To  compare  two  discrete 
distributions  we  define  first  d(u)  and  then  D(u)  as  follows: 

d(u)  =  d(u-,H,F)  = 

D(u)  =  /  d(t)dt. 

J  0 

We  apply  this  definition  to  the  discrete  sample  distributions  F A  and  F“  k  to  obtain 

=  d(u-,r,Fk-) 

and  its  integral  Dk{u). 

We  obtain  the  following  definition  of  dk“(u)  for  the  c  sample  testing  problem  with  all 
values  distinct: 

dk~{u)  =  N/nk  if  (JSfc(j)  -  1  )/N  <u<  Rk[j)/N,j  =  1,. . . 

=  0,  otherwise. 

A  component,  with  score  function  J{ u),  is  a  linear  functional 

TkV)  -  F 

JO 

It  equals 

*  /•**(>)/* 

(!/»*)  L0 

yTi  J{Rh(})-1)/N 

which  can  be  approximated  by  £T[J(PA(i%))]. 

LINEAR  RANK  STATISTICS.  The  concept  of  a  linear  rank  statistic  to  compare 
the  equality  of  c  samples  does  not  have  a  universally  accepted  definition.  One  possible 
definition  is  ^ 

AV)  =  (l/rtfc)  £  J((R*U)  -  .5)/J V) 

;=1 

However  we  choose  the  definition  of  a  linear  rank  statistic  as  a  linear  functional  of  dfc~(u), 
which  we  call  a  component;  it  is  approximately  equal  to  the  above  formula. 

We  define 


TkV)  =  m  -  x)  VAR[J(tr)|p.*/(J  f  J(u){dkV)  -  !><<<* 

Jo 


(!) 


where  U  is  Uniform{0,  l},  E\J(U) J  =  Jq  J(u)du, 

VAR [J(C/)]  =  /  {J(u)  -  E[J{U)]}2du. 

Jo 

Note  that  the  integral  in  the  definition  of  Tk“(J)  equals 

f  J{u)d{Dh~[u)  -  u}. 

Jo 
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The  components  of  the  Kruskal- Wallis  nonparametric  test  statistic  TKW2  for  testing 
the  equality  of  c  means  have  score  function  J[u)  —  u  -  .5  satisfying 

E[J{U)]  =  .5,  VAR[J(!7)]  =  1/12. 

The  components  of  F  test  statistic  T2  have  score  function 

J(u)  -  {Q»  -  Y‘)la 

where  Q*{u)  is  sample  quantile  function  of  the  pooled  sample  Y . 

IS.  GENERAL  DISTANCE  MEASURES.  General  measures  of  the  distance  of  D"(u) 
from  u  and  of  cf(u)  from  1  are  provided  by  the  integrals  from  0  to  1  of 

{<*»  - 1}2,  {£(»  -  u}2,  u}J/«(i-«),  {<r(u)  -  i}s 

where  <T(u)  is  a  smooth  version  of  <T(u).  We  will  see  that  these  measures  can  be  decom¬ 
posed  into  components  which  may  provide  more  insight;  recall  basic  components  are  linear 
functionals  defined  by  (!) 


Jo 

If  0i(u),  i  =  0, 1, 2, ...»  are  complete  orthonorraal functions  with  <f> o  =  1,  then  Hq  can 
be  tested  by  diagnosing  the  rate  of  increase  (as  a  function  of  m  »  1,2, . . .)  of 

A<M«)  - 1>»*  -  Dr(*l* 

i- 1 

which  measure  the  distance  from  1  of  the  approximating  smooth  densities 

4W-£rw*,(.). 

»- 1 

16.  ORTHOGONAL  POLYNOMIAL  COMPONENTS.  Let  Pi(x)  be  Legendre  poly¬ 
nomials  on  (-1,1); 

Pl(*)  -  * 

p2(x)  =  (3x2  -  l)/2, 
p3(*)  =  (5x3-3*)/2, 

P4(x)  =  35x4  -  30*2  +  3. 

Define  Legendre  polynomial  score  functions 

<t>Li(u)  =  (2*  +  l)‘5Pi(2u  -  1). 

One  can  show  that  an  Anderson-Darling  type  statistic,  denoted  AD(D')1  can  be  repre¬ 
sented 

AD(D“ )  =  /  {(D"(u)  -  u}2/u(l  --  u)}du 
Jo 

=  f;ir(^)|2/(t(*  +  i)) 

»=1 
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Define  cosine  score  functions  by 

<f>C{[u)  =  2'5e0B(t7ru). 

One  can  show  that  a  Cramer-von  Mises  type  statistic,  denoted  CM(D "),  can  be  repre¬ 
sented 

CM{D")  *  ['{D'iv)  -  u}2du 

Jo 

-JirwCiJiVt.V)* 

t-1 

In  addition  to  Legendre  polynomial  and  cosine  components  we  consider  Hermite  poly¬ 
nomial  components  corresponding  to  Hermite  polynomial  score  functions 

=  (<l)--5iri(«rl(<.)) 

where  Hi(x)  are  the  Hermite  polynomials: 

Hi(x)  -  x , 

J?2(*)  =  x2  -  1, 

H$(x)  =  x3  -  3x, 

Hi{x)  =  a4  -  6x2  +  3. 

17.  QUARTILE  COMPONENTS  AND  CHI-SQUARE.  Quartile  diagnostics  of  the 
null  hypothesis  Hq  are  provided  by  component*  with  quartile  “square  wave"  score  functions 

SQi(w)  =  -25,  0  <  it  <  .25, 

=  0,  .25  <  tt  <  .75, 

=  2'8,  .75  <  u  <  1; 

5Qa(u)  =  1*  0  <  u  <  .25, 

—  -1,  .25  <  u  <  .75, 

as  1,  .75  <  u  <  1; 

SQz(u)  =0  if  0  <  u  <  .25  or  .75  <  u  <  1, 

=  —  2*®,  .25  <  u  <  .5, 

=  2‘5,  .5  <  u  <  ,75. 

A  chi-squared  portmanteau  statistic,  which  is  chi-squared (3),  is 

3 

cQk  -  (w-  ljpjk/d  -  p.k)  £  |r(sg,)|J 

~{N  -  l)p.*/(l  -  p.k)  /  Wjt(u)  -  i-)2du 

Jo 

defining  the  quartile  density  (for  i  —  1,2, 3,4) 

dQfc(u)  =  4{ZV(t(.25))  -  Dk{(i  -  l).25),(i  -  1).25  <  u  <  i(.25) 
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A  pooled  portmanteau  chi-squared  statistic  is 

CQ  =  til  -  ?.k)CQk 

Jk«l 

18 .  DIVERSE  STATISTICS  AVAILABLE  TO  TEST  EQUALITY  OF  e  SAMPLES. 
The  problem  of  statistical  infereence  is  not  that  we  don’t  have  answers  to  a  given  question; 
usually  we  have  too  many  answers  and  we  don’t  know  which  one  to  choose.  A  unified 
framework  may  help  determine  optimum  choices.  To  compare  c  samples  we  can  compute 
the  following  functions  and  statistics: 
ll  comparison  densities:  <V(u), 

21  comparison  distributions  (u), 

3)  quartile  comparison  density  dQk(u),  quartile  density  chi-square 

CQk  ~  —  1)p.Aj/ (1  “  P.k )  f  {^Qfe(u)  “  l}2du. 

JO 

4)  non-parametric  regression  smoothing  of  dfc~(u)  using  a  boundary  Epanechnlkov  kernel, 
denoted  <V(u), 

5)  Legendre  components  and  chi-squares  up  to  order  4  are  defined  using  definition  (1)  of 
T k  • 

TLk(t)  *  TMLi) 

»-l 

CL{m)  m  £(1  -P.k)CLh{m) 
fc-1 

ADh^f^\TLk{t)\2/i(i  +  l) 

a 

AD  =  J2^~P.I,)ADk 
fcs*l 

6)  Cosine  components  and  chi-squares  up  to  order  4  are  defined: 

TCM  -  JVWCi) 

cct(m)=f;irc*(i)lJ 

»*1 

c 

CC[m)  =  5^(1  -  P'k)CCk(m) 
k=l 

CMk  -  2  |TCfc(i)|2/(*>)2 
»*1 

CM=^(1  -  p.k)CMk 
k=  l 
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7)  Hermite  components  and  chi-squares  up  to  order  4  are  defined: 

THk[i)  =  r/H) 

m 

CHk(m)  =  £  IrfftWI2 

»-l 

CH(m)  =  £(1  -  p,k)CHk(m) 

Jb-1 

8)  density  estimators  d**(tt)  computed  from  components  up  to  order  4, 

9)  entropy  measures  with  penalty  terms  which  can  be  used  to  determine  how  many 

components  to  use  in  the  above  test  statistics 

19.  EXAMPLES  OF  DATA  ANALYSIS .  The  interpretation  of  the  diversity  of  statis¬ 
tics  available  is  best  Illustrated  by  examples. 

In  order  to  compare  our  methods  with  others  available  we  consider  data  analysed  by 
Boos  (1986)  on  ratio  of  assessed  value  to  sale  price  of  residential  property  in  Fitchburg, 
Mass.,  1979.  The  samples  (denoted  I,  II,  III,  IV)  represent  dwellings  in  the  categories 
single-family,  two-family,  three-family,  four  or  more  families.  The  sample  sizes  (54,  43, 
31,  28)  are  proportions  .346,  .276,  .199,  .179  of  the  size  156  of  the  pooled  sample.  We 
compute  Legendre,  cosine,  Hermite  components  up  to  order  4  of  the  4  samples;  they  are 
asymptotically  standard  normal.  We  consider  components  greater  than  2  (3)  in  absolute 
value  to  be  significant  (very  significant). 

Legendre,  cosine,  and  Hermite  components  are  very  significant  only  for  sample  I, 
order  1  (-4.06,  -4.22,  -3.56  respectively).  Legendre  components  are  significant  for  sample 
IV,  orders  1  and  2  (2.19,  2.31).  Cosine  components  are  significant  for  sample  IV,  orders  I 
and  II  (2.36,  2.23)  and  sample  III,  order  1  (2.05).  Hermite  components  are  significant  for 
sample  IV,  orders  2  and  3  (2.7  and  -2.07). 

Conclusions  are  that  the  four  samples  are  not  homogeneous  (have  the  same  distribu¬ 
tions).  Samples  I  and  IV  are  significantly  different  from  the  pooled  sample.  Estimators 
of  the  comparison  density  show  that  sample  I  is  more  likely  to  have  lower  values  than  the 
pooled  sample,  and  sample  IV  is  more  likely  to  have  higher  values.  While  all  the  statistical 
measures  described  above  have  been  computed,  the  insights  are  provided  by  the  linear  rank 
statistics  of  orthogonal  polynomials  rather  than  by  portmanteau  statistics  of  Cramer-von 
Mises  or  Anderson-Darling  type. 

SO.  CONCLUSIONS,  The  goal  of  our  recent  research  (see  Parzen  (1979),  (1983)) 
on  unifying  statistical  methods  (especially  using  quantile  function  concepts)  has  been  to 
help  the  development  of  both  the  theory  and  practice  of  statistical  data  analysis.  Our 
ultimate  aim  is  to  make  it  easier  to  apply  statistical  methods  by  unifying  them  in  ways 
that  increase  understanding,  and  thus  enable  researchers  to  more  easily  choose  methods 
that  provide  greatest  insight  for  their  problem.  We  believe  that  if  one  can  think  of  several 
ways  of  looking  at  a  data  analysis  one  should  do  so.  However  to  relate  and  compare  the 
answers,  and  thus  arrive  at  a  confident  conclusion,  a  general  framework  seems  to  us  to  be 
required. 

One  of  the  motivations  for  this  paper  was  to  understand  two-sample  tests  of  the 
Anderson-Darling  type;  they  are  discussed  by  Pettitt  (1976)  and  Scholz  and  Stephens 
(1987).  This  paper  provides  new  formulas  for  these  test  statistics  based  on  our  new  def¬ 
inition  of  sample  comparison  density  functions.  Asymptotic  distribution  theory  for  rank 
processes  defined  by  Parzen  (1983)  is  given  by  Aly,  Csorgo,  and  Horvath  (1987);  an  excel¬ 
lent  review  of  theory  for  rank  processes  is  given  by  Shorack  and  Wellner  (1986). 


23 


However  one  can  look  at  k  sample  Anderson-Darling  statistics  as  a  single  number 
formed  from  combining  many  test  statistics  called  components.  The  importance  of  com¬ 
ponents  is  also  advocated  by  Boos  (1986),  Eubank,  La  Riccia,  and  Rosenstein  (1987)  and 
Alexander  (1989).  Insight  is  greatly  increased  if  instead  of  basing  one’s  conclusions  on 
the  values  of  single  test  statistics,  one  looks  at  the  components  and  also  at  graphs  of  the 
densities  of  which  the  components  are  linear  functionals  corresponding  to  various  score 
functions.  The  question  of  which  score  functions  to  use  can  be  answered  by  considering 
the  tail  behavior  of  the  distributions  that  seem  to  fit  the  data. 
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For  samples  I  and  IV,  sample  comparison  distribution  function  D  (u) 


i 
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For  samples  I  and  IV,  sample  comparison  density  <f(u),  sample  quartile  density  dQ'[u ) 
(square  wave),  nonparametric  density  estimator  cf(u) 
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For  samples  I  and  IV,  Legendre,  cosine,  and  Hermite  orthogonal  polynomial  estimator  of 
order  4  of  the  comparison  density,  denoted  d^(u),  compared  to  sample  quartile  density 
dQ~(u). 

Ity,  Cos(x*s),  Hire's)  hnsily  Its,  Cos(x's),  s)  D*nsity 


Reliability  of  the  M256  Chemical  Detection  Kit 


David  W.  Webb  &  Linda  L,  C.  Moss 
U.S.  Army  Ballistic  Research  Laboratory 


Abstract 

The  U.S.  Army  uses  the  M256  Chemical  Detection  Kit  (CDK)  to  indicate  the  presence 
or  absence  of  certain  agents  In  the  battlefield,  which  is  indicated  by  a  color  change  on  the  kit. 
Strength  of  response  is  also  influenced  by  the  quantity  of  agent.  Lots  must  meet  reliability 
specifications  to  be  considered  "battle-ready".  How  do  we  go  about  collecting  and  analyzing 
our  data  so  as  to  evaluate  its  reliability?  Other  problems  of  interest  include  quantifying  how 
the  agent  quantity  affects  the  response  and  if  there  are  differences  between  the  two  manufac¬ 
turers  of  the  M256  CDK.  Consultants  at  the  Ballistic  Research  Laboratory  have  employed  a 
dose-response  framework  to  study  the  reliability  problem.  We  use  a  binary  response 
(present/not  present)  and  assume  a  lognormal  distribution  in  arriving  at  a  response  curve  for 
each  lot.  Assessments  of  our  approach  and  suggestions  for  alternative  approaches  are  asked 
of  the  panel. 
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Description  of  Kit 


Hie  M256  Chemical  Detection  Kit  (CDK)  is  used  to  detect  the  presence/absence  of 
dangerous  concentrations  of  toxic  agents  by  color- changing  chemical  reactions.  Each  CDK 
contains  twelve  samplers,  which  are  the  actual  testing  devices.  Four  types  of  agents  can  be 
detected  with  the  CDK.  The  tests  indicate 

a)  if  it  is  permissible  to  remove  the  protective  mask  following  an  agent  attack, 

b)  if  agent  is  present  in  the  air  or  on  surfaces  suspected  of  contamination, 

c)  if  any  agent  is  present  after  decontamination  operations. 

The  U.S.  Army  requires  that  the  samplers  exhibit  at  least  a  92.5%  reliability  (with  90% 
confidence)  in  responding  to  agent  concentrations  at  the  specification  levels.  However,  the 
kit  should  not  be  so  sensitive  that  soldiers  wear  their  mask  at  safe  levels  of  concentration, 
thereby  interrupting  other  battlefield  duties. 

On  the  back  of  each  sampler  are  complete  instructions  for  testing  and  colored  examples 
of  safe  and  danger  responses.  After  performing  the  test,  a  paper  test  spot  is  checked  for  any 
change  of  color.  The  color  change  will  not  usually  be  an  exact  match  with  the  colors  shown 
on  the  back  of  the  sampler.  This  is  because  the  response  depends  upon  the  agent  quantity. 
To  make  matters  more  complex,  when  the  agent  is  present  the  observed  response  may  be 
nonuniform  with  a  few  shades  of  the  danger  response  showing. 


Test  Conditions  &  Restrictions 


The  lots  of  kits  differ  in  manufacturer  (A  or  B),  age  (1-8  years),  and  storage  site  (8  sites 
in  the  United  States  and  Europe).  Not  all  combinations  of  these  three  factors  are 
represented  in  the  design  matrix;  in  fact,  the  design  matrix  is  very  sparse.  For  example,  there 
was  only  one  lot  that  was  eight  years  old. 

Most  lots  contain  ten  or  more  kits  (therefore,  120  or  more  individual  samplers).  Some 
lots  contained  as  many  as  1000  kits,  while  others  had  as  few  as  one  kit. 

We  are  restricted  to  the  number  of  samplers  that  may  be  tested  at  any  time  since  the 
test  chamber  is  large  enough  to  hold  only  six  samplers.  Another  restriction  lies  in  the  fact  that 
testing  laboratories  are  only  available  for  the  length  of  time  designated  in  the  work  contract. 
This  usually  is  no  more  than  two  months. 
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The  test  equipment  that  controls  the  concentration  of  agent  in  the  test  chamber  is  very 
accurate  and  precise,  but  it  is  slow.  It  may  take  about  an  hour  to  change  to  a  higher  concen¬ 
tration  When  going  from  a  high  to  a  low  concentration,  the  waiting  period  may  be  several 
hours  since  the  high  concentration  tends  to  leave  a  residual  amount  of  agent  in  the  test 
chamber. 


Our  Approach 

We  have  decided  to  evaluate  each  agent  and  the  chosen  lots  separately.  From  each 
manufacturer,  we  have  selected  one  lot  from  the  available  age  groups.  Also,  we  have  tried  to 
choose  lots  of  similar  age  from  the  manufacturers  so  that  they  can  be  paired  and  we  can  look 
for  general  trends.  In  all,  we  have  chosen  fifteen  lots  ranging  in  age  from  1  to  8  years. 
Although  the  sites  are  in  varying  climatic  areas,  most  of  the  warehouses  are  humidity  and 
temperature  controlled;  therefore  the  locations  are  treated  as  homogenous.  Differences 
existing  between  manufacturers  are  not  considered  in  our  initial  design,  but  will  be  addressed 
later. 

We  have  taken  the  route  of  estimating  the  reliability  of  each  lot  at  the  specification  level 
of  each  agent.  We  have  also  chosen  a  dose-response  type  experiment,  where  our  dose  is  the 
agent  concentration  and  the  response  is  safe/danger.  For  the  purpose  of  determining 
response,  U.S.  Army  manuals  specify  a  set  of  nine  color  chips  that  progressively  range  from 
the  "safe"  color  to  the  "danger"  color.  The  manual  also  states  a  cutoff  color  for  the  Bernoulli 
response.  (In  most  cases,  color  chips  1-3  correspond  to  a  safe  response,  while  chips  4-9  are 
considered  danger  responses.) 

We  have  made  the  assumption  that  the  response  curves  follow  that  of  the  lognormal 
cumulative  distribution  function  with  unknown  mean  and  standard  deviation.  The  lognormal 
was  selected  based  on  historical  precedent,  although  we  note  that  the  log-logistic  would  have 
also  been  a  reasonable  choice. 

To  choose  the  concentration  levels  at  which  to  run  the  tests,  we  have  considered  several 
candidate  sequential  designs.  In  light  of  some  of  our  restrictions,  however,  none  of  these 
would  be  very  practical  (e.g.,  Robbins-Monro  would  have  required  too  much  laboratory 
time). 

Instead,  we  have  chosen  a  two-stage  "semi"-fixed  design.  In  the  first  stage,  11  samplers 
are  tested  at  seven  different  levels;  one  concentration  level  set  at  an  estimated  mean,  three 
concentrations  above  this  estimated  mean,  and  three  concentration  levels  below  the 
estimated  mean,  each  being  a  multiple  of  the  standard  deviation  away  from  the  mean.  Mean 
and  standard  deviation  estimates  are  based  on  the  results  of  a  pretest  (which  for  the  purpose 
of  brevity  is  deleted  from  this  presentation).  The  multiple  of  the  standard  deviation  is  chosen 
so  that  the  specification  level  will  be  covered  by  the  seven  test  concentrations. 
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Stage  I 

Concentration  Number  of  Samplers 


Aj  ■  3kdj  1 

A1-2k^l  1 

At  -  2 

Ai  3 

Ai  +  kd.  2 

Aj  +*  2kdj  1 

Aj  +  3kdj  1 

11 

Note:  k  is  chosen  so  that  the  seven  test  concentrations  cover  the  specification  level.  Ai  and 
come  from  the  pretest. 

At  the  conclusion  of  Stage  I,  the  data  are  analyzed  using  the  DiDonato- J arnagin  max¬ 
imum  likelihood  estimation  algorithm  to  produce  new  estimates  of  the  parameters,  A2  and 
&2 .  In  Stage  II,  nine  more  units  are  tested  at  five  concentration  levels;  one  level  set  at  the 
new  estimated  mean,  and  at  two  levels  above  and  below  this,  each  now  being  a  multiple  of  the 
new  standard  deviation  from  the  mean. 


Stage  II 

Concentration  Number  of  Samplers 

1 
2 
3 
2 
1 
9 

At  the  conclusion  of  Stage  II,  the  parameter  estimates  for  the  lot  are  re-evaluated  using 
all  20  data  points,  giving  us  a  final  A  and  d.  With  these  final  estimates,  the  .925  quantile  is 
estimated  by  A  +  z(  925)  d. 

By  taking  the  variance  of  the  above  equation,  we  get  an  estimate  of  the  variance  of  the 
.925  quantile, 


^2 

fii  +  A 


Var (ft)  +  (J,  ms/  Var(9)  +  2  z(S23)  Cov(S, ») 

(The  DiDonato-Jamagin  algorithm  gives  the  values  of  the  variances  and  covariance  term.)  If 
the  one-sided  90%  upper  confidence  limit  of  the  .925  quantile  is  less  than  the  specification 
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concentration,  then  we  can  conclude  that  the  lot  meets  the  requirement  for  that  particular 
agent. 

We  do  not  have  a  statistical  technique  per  se  for  detecting  significant  differences 
between  manufacturers  or  sites.  Our  "approach"  would  be  to  simply  look  for  any  obvious 
trends  or  differences.  To  study  the  age  issue,  a  separate  accelerated  life  test  will  be  con¬ 
ducted  at  a  later  date. 


Questions 

1.  Is  our  approach  appropriate  for  determining  an  extreme  quantile? 

2.  Can  one  estimate  a  quantile  when  considering  more  than  two  possible  responses  (e.g., 
the  nine  color  chips)? 

3.  How  might  we  statistically  compare  the  reliability  of  the  manufacturers  (or  sites)? 


Concluding  Remarks 

Following  our  presentation,  we  heard  comments  and  suggestions  from  the  clinical  ses¬ 
sion  panelists  and  audience.  Two  major  concerns  were  expressed  by  several  persons.  First 
was  uneasinessd  towards  our  assumption  of  a  log-normal  distribution.  Some  respondents  felt 
this  to  be  a  potentially  dangerous  assumption,  especially  since  we  are  estimating  the  tail  of 
our  distribution.  Secondly,  some  persons  questioned  our  method  of  estimating  the  mean  of 
the  distribution,  and  then  extrapolating  to  the  .925  quantile.  These  two  problems  could  lead 
to  some  very  erroneous  conclusions. 

In  general,  the  comments  we  heard  confirmed  our  beliefs  that  this  is  a  very  difficult 
problem  to  analyze,  in  light  of  the  small  sample  sizes  and  other  laboratory  constraints  to 
which  the  test  is  subjected.  Although  no  definitive  alternative  approaches  arose  rrom  our  dis¬ 
cussions,  some  possible  attacks  that  were  suggested  to  us  included  - 

1.  Sampling  more  towards  the  tails  of  the  distribution. 

2.  Isotonic  regression. 

3.  Testing  at  the  specification  level  and  employing  a  general  linear  model  approach 
with  the  color  chip  number  corresponding  to  the  color  change  as  the  response  and 
age,  manufacturer,  and  storage  site  as  variables. 

We  would  like  to  thank  the  panelists  and  audience  for  their  many  suggestions  and 
remarks. 
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COMPARISON  OF  RELIABILITY  CONFIDENCE  INTERVALS 


Paul  H.  Thrasher 
Engineering  and  Analysis  Branch 
Reliability,  Availability,  and  Maintainability  Division 
Army  Materiel  Test  and  Evaluation  Directorate 
White  Sands  Missile  Range,  New  Mexico  88002-5175 

ABSTRACT 

Some  confidence  intervals  on  reliabilities  are  investigated.  Only 
binomial  events  are  considered.  Only  the  narrowest  two-sided  and  the  upper 
one-sided  confidence  intervals  are  calculated.  Three  methods  of  estimating 
the  distribution  of  reliabilities  are  reviewed  and  compared.  These  are  the 
Fisherian  approach,  the  Bayesian  approach  with  the  ignorant  prior,  and  the 
Bayesian  approach  with  the  noninformed  or  noninformative  binomial  prior.  Both 
the  width  and  location  of  the  confidence  intervals  differ  for  these  three 
methods. 


INTRODUCTION 

Reliability  estimates  are  not  as  straightforward  as  might  be  expected. 
Measurement  of  a  numher  of  successes  x  in  a  sample  size  n  quickly  leads  to  a 
point  estimate  of  the  reliability  R  equal  to  x/n.  Estimates  of  confidence 
intervals  are  more  difficult  to  obtain  however.  Two  things  in  addition  to  the 
data  are  needed  for  confidence  interval  estimation.  First,  some  function  must 
be  used  to  describe  the  reliabilities.  Second,  a  method  must  be  selected  to 
locate  the  confidence  interval  within  the  function. 

The  purpose  of  this  paper  is  to  compare  various  functions  describing 
reliabilities.  For  simplicity,  all  tested  items  will  be  assumed  dichotomous 
and  Independent.  That  is,  the  binomial  b(x;n,R)  is  assumed  to  describe  the 
random  variable  x  if  n  and  R  are  known.  The  problem  is  to  select  a  function 
for  R  when  x  and  n  are  known.  The  three  functions  considered  here  are  based 
on  (1)  the  Fisherian  approach  and  (2)  the  Bayesian  technique  using  prior 
distributions  of  R  when  (A)  R  is  equally  likely  to  be  any  value  between  zero 
and  one  and  (B)  R  is  unknown  numerically  but  it  is  known  to  be  a  binomial 
parameter . 

To  focus  attention  on  the  comparison  of  the  confidence  intervals  from 
these  three  functions,  the  methods  used  to  locate  the  confidence  intervals  are 
restricted  in  this  paper.  Only  two  methods  are  used  in  calculations;  one  is 
one-sided  and  the  other  is  two-sided. 
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The  one-sided  confidence  interval  considered  is  the  upper  confidence 
Interval.  This  is  based  on  the  premise  that  having  a  reliability  too  low  is 
much  more  serious  than  the  reliability  being  toe  high. 

The  two-sided  confidence  Interval  considered  is  the  narrowest  possible 
(Rankin).  This  is  illustrated  In  Figure  1.  It  is  located  by  adjusting  the 
confidence  limits  until  (1)  the  sum  of  the  areas  under, the  tails  is  a  and  (2) 
the  functions  of  these  two  limits  are  equal.  This  correspondence  of  narrowest 
interval  with  equal  heights  Is  a  geometrical  property.  It  Is  not  based  on  the 
choice  of  the  function  describing  R.  It  may  be  demonstrated  by  (1)  starting 
with  the  confidence  limits  of  points  equal  heights,  (2)  moving  the  left 
confidence  limit  to  the  right,  and  (3)  noting  that  the  right  limit  has  to  be 
moved  further  to  the  right  in  order  to  keep  the  sum  of  the  areas  under  the 
tails  constant.  This  Is  shown  In  Figure  1  by  the  dashed  lines.  A  similar 
argument  starts  by  moving  the  narrowest  confidence  limit  to  the  left. 

Other  possible  two-sided  confidence  limits,  not  calculated  In  this  paper, 
are  illustrated  in  Figures  2  and  3.  These  are  the  traditional  equal-dlvlslon- 
of-area-under-the-tails  interval  and  the  maximum-1 1 kel Ihood-estlmator- In-the- 
center  interval.  The  first  Is  the  easiest  to  calculate.  The  second  has  a 
symmetric  appeal  but  It  Is  non-exlstant  when  the  peak  of  the  curve  Is  not  at 
R-0.5  and  l'-0  Is  sufficiently  large. 

FISHERIAN  APPROACH 

The  traditional  Fisherian  approach  (Mann,  Shafer,  and  Slngpurwalla) 
considers  sums  of  binomial  probabilities.  This  approach  yields  two  Beta 
functions.  The  lower  confidence  limit  Is  obtained  from  one  Beta  function;  a 
second  function  is  needed  for  the  upper  limit. 

Lower  Confidence  Limit: 

The  lower  ( 1  -  a )  1 00%  confidence  limit  R_  is  defined  by  P[R>Fp*l-a.  An 
alternate  expression  Is  P[R<R]"a.  The  limit  _R  is  the  largest  value  of  R  that 
makes  the  data  x  and  n  plausible.  Plausibility  is  defined  as  satisfaction  of 
the  degree  of  confidence  l-a  of  correctly  selecting  the  right  R_.  The  lower 
100%  confidence  limit  of  R  Is  R_«0  because  all  values  of  R  satisfy  R>0. 
Increasing  .R  requires  a  decrease  In  l-a  or  an  Increase  In  0.  This  Increase  In 
R.  shifts  the  binomial  distribution  of  the  possible  measurements  i  which 
resulted  in  the  single  measurement  x.  For  the  limitir.n  case  of  R»0,  the 
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binomial  b(i;n,R)  consists  of  a  single  spike  of  unit  height  at  1*0.  As  £  and 
a  Increase,  b(1;n,R)  takes  a  shape  Illustrated  in  Figure  4  and  described  by 

b(1;n,R)  -  Q  R 1  (l-R)"’1 

where  the  number  of  ways  of  obtaining  1  successes  In  n  trials  If  found  from  by 

/n  \  n!  _  n(n-l) . . .( n- 1+1 ) 

\ 1  /  1!  (n-1) I  "  1(1-1). ..(1)  ’ 

The  extent  of  the  shifting  from  the  single  spike  Is  determined  by  the  data 
x  and  n.  The  value  of  £  Is  determined  In  two  steps.  First,  £  Is  Increased 
until  the  sum  b(x;n,R)+b(x+ljnfiR)+.  ..+b(n;n  ,R)  equals  the  probability  a  making 
the  confidence  relation  P[R<RJ*o  or  P[R>R]*l-0  untrue.  Second,  the  continuous 
variable  R_  is  decreased  Infinitesimally  making  the  confidence  relation 
P[R>Rj"l-a  just  barely  valid.  Thus  R  is  neither  too  large  or  too  small  to  be 
a  (l-o)100X  lower  confidence  limit  on  R  when 


The  extraction  of  R_  from  this  equation  can  be  facilitated  by  using  a  Beta 
function  as  described  In  the  following  paragraph.  Before  doing  that  however, 
It  Is  expedient  to  note  that  a  measurement  of  x«0  implies  that  jl-0  for  all 
values  of  a*  This  special  case  Isn't  algebraically  Included  In  the  following 
Beta  function.  It  Is  adroitly  described  by  an  argument  based  on  Figure  4: 
when  x«0,  R  has  to  be  0  to  make  b{0;n,RJ>l  and  b(1;n,Rj*0  for  all  i#0. 

The  Beta  function  of  R  is 


f(R) 


r(a+b) 

r(a)r(b) 


R3"1  ( 1 -R ) b” 1 


where  a  and  b  are  parameters.  Using  the  equality  of  the  gamma  function  r(j) 
and  the  factorial  (j-l)l  when  j  Is  an  integer  yields 


f(R) 


( a+b-1 )  I 
( a-1) ! (b-1) ! 


Ra-i  ( i .r ) b- 1 ^ 
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Postulating  that  the  reliability  is  described  by  f(R)  and  setting  the  area  to 
the  left  of  R  at  a  yields 


R 

a  -  f  f(R)  dR. 
0 


Repeated  Integrations  by  part  yield 

o  -  *\b'1  ^a+b*^  i1  (l-R)a+b'1“1‘ 

Comparison  of  this  summation  and  the  summation  for  a  in  the  previous  paragraph 
yields  a-x  and  a+b-l“n.  Thus  the  parameters  in  the  Beta  function  for  the 
lower  limit  _R  are  a-x  and  b*»n+l-x. 

Upper  Confidence  Limit: 

The  upper  confidence  limit  R  defined  by  P[R<R,]*l“a  Is  obtained  from 
another  Beta  function.  Arguments  similar  to  those  In  the  proceeding  section 
yield  the  upper  Beta  function  In  four  steps: 

(1)  R  Is  In  the  binomial  sun 


i.o  W 


(2)  IT  Is  the  lower  limit  of  integration  over  the  second  Beta  function 

a  ■  f f(R)  dR, 

R 

(3)  repeated  Integrations  by  parts  transform  this  Integral  to  the  summation 


a  »  V1  (*'**' ^  (l-R)a'+b''1‘1.  and 

1-0  \ 


1 


(4)  the  second  Beta  function  parameters  are  identified  by  x* a ' -1  and 
n-a'+b'-l  to  be  a'-x+l  and  b'-n-x. 
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This  Beta  function  does  not  describe  R  when  x=n  because  r(b' )Br(0)«(0-l) 1 
is  meaningless.  For  this  special  case,  R»1  for  all  0.  This  may  be  seen  from 
a  binomial  distribution  symmetric  to  Figure  4.  Using  an  R:  near  1  and  an  a 
containing  binomial  terms  from  j«0  to  j«x«n,  it  is  easily  seen  that  «  Is  1 
even  when  R  Is  1.  Since  *R  continuous,  R»1  for  any  value  of  l-0. 

BAYESIAN  APPROACH 

The  Bayesian  approach  (Martz  wj  Waller)  uses  the  data  x  and  n  to  update 
a  prior  distribution  g(R)  describing  R  to  a  posterior  distribution  g(R|x) 
describing  R  after  x  Is  given.  The  algebraic  relation  between  these  two  Is 
based  on  the  equality  of  the  joint  density  h(x,R)  to  both  the  product 
g(R | x)f ( x)  and  the  product  f(x|R)g(R).  Thus  the  posterior  Is  found  from 

g(R|x)  -  f(x|R)  g(R)  /  f(x). 

This  expression  is  simplified  by  noting  that  (1)  the  conditional  density  of  x 
given  R  Is 


f ( x | R )  -  b(x;n,R)  «  ^  Rx  (1-R)n‘x 

and  (2)  the  marginal  density  f(x)  from  the  Integral  of  h(x,R)-f(x | R)g(R)  Is 

1  1  /n\ 

f(x)  -  f  f ( x | R )  g(R)  dR  -  /  ^ JR*  (1-R)n"x  g(R)  dR. 


Thus  the  general  posterior  Is 


R*  (1-R)n“*  g(R) 

g(R|X)  -  - 

J1  Rx  (1-R)n_x  g(R)  dR 
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Ignorant  Prior; 

One  prior  that  can  be  used  is  the  uniform  distribution  g(R)»l  for  0<R<1 
and  g(R)»0  elsewhere.  This  is  sometimes  called  the  Ignorant  prior  because  all 
values  of  R  between  0  and  1  are  equally  likely.  That  Is,  there  is  no  evidence 
to  favor  the  selection  of  any  value  of  R  over  any  other  R  between  0  and  1. 

Use  of  this  prior  in  the  general  posterior  yields 


9(R|x) 


R*  (1-R)n“x 

1 

/  R*  (1-R)n"x  dR 


Integration  by  parts  evaluates  the  denominator.  The  posterior  is  thus 

r( n+2) 

g(R|x)  - -  r(*+1)-1  (1-r)("-x+D-1. 

r( x+1) r(n-x+l) 

This  is  a  Beta  function  with  parameters  a*x+l  and  b«n-x+l. 

Noninformed  Prior: 

A  second  prior  that  can  be  used  recognizes  that  the  reliability  Is  a 
binomial  parameter  but  has  no  Information  about  Its  value.  This  Is  sometimes 
called  the  noninformed  or  nonlnformatlve  binomial  prior. 

Every  noninformed  prior  is  based  on  a  transformation  making  the 
probability  density  Insensitive  to  the  data.  For  the  binomial  parameter  R  In 
b(x;n,R),  It  has  been  empirically  found  (Box  and  Tlao)  that  plots  of 
K(x,n)b(x;n,$)  versus  $  yield  very  similar  curves  for  fixed  n  and  different 
x's  when  (1)  K(x,n)  is  determined  by  numerical  Integration  to  make  the  area 
under  K(x,n)b(x;n,^)  equal  to  one  and  (2)  $  is  given  by 

$  ■  Arcs1n(R1//2 )  * 


Figures  5  and  6  show  that  for  0<x<n  these  similar  curves  become  nearly  equally 
spaced  along  the  axis  as  n  Is  Increased.  The  noninformed  argument  assumes 
that  all  n+1  curves  are  essentialy  equal  and  equally  spaced  for  all  n.  This 
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makes  being  non  In  formed  about  x  equivalent  to  being  Ignorant  about  <j>.  The 
prior  assumption  that  (1)  x  Is  unknown  but  (2)  the  situation  Is  described  by 
the  one  of  these  curves  thus  leads  to  a  prior  distribution  of  $  that  is 
uniform  between  4,0*  and  4»90* .  The  corresponding  prior  of  R  may  be  found 
from  the  transformation  of  variable  technique  (Freund  and  Walpole)  by  applying 


g(R) 


Using  h( 4, ) -1  and  sin^)"^2  in  this  equation  yields  g(R)Bl/{2[R(l-R)]1/2}. 
Use  of  this  binomial  noninformed  prior  in  the  general  posterior  yields 


g(R|x) 


rX-1/2  (1.R )n-x-l/2 


j  r(x+1/2)-1  (i-R)(n-x+l/2)-l  Jr 

0 


The  denominator  Is  recognized  as  an  Integral  over  a  Beta  function.  It  Is 
evaluated  to  be  r( x+1/2) r( n-x+1/2) /r( n+1) .  The  posterior  is  thus  found  to  be 
a  Beta  function  with  a- x+1/2  and  b«n-x+l/2. 

COMPARISON  OF  CONFIDENCE  INTERVALS 

The  three  methods  reviewed  In  the  previous  sections  have  been  applied  to 
confidence  Intervals  on  reliability.  Both  two-sided  and  one-sided  Intervals 
have  been  Investigated. 

Narrowest  Two-Sided  Intervals: 

Figures  7  through  15  show  distributions  and  narrowest  two-sided  80% 
confidence  Intervals.  Figure  7  Illustrates  the  symmetry  about  x»n/2.  Thus 
graphs  for  x<n/2  are  not  needed  to  investigate  trends.  Figure  8  Is  one 
example  of  the  destruction  of  symmetry  by  making  x>n/2.  Figure  9  shows  that 
when  x»n  the  symmetry  Is  so  completely  destroyed  that  the  narrowest  two-sided 
Intervals  are  actually  upper  one-sided  intervals.  Figures  10,  11,  and  12  and 
Figures  13,  14,  and  15  show  the  effect  of  increasing  n:  for  fixed  x,  the 
confidence  intervals  all  become  narrower  but  the  relationship  of  the 
Fisherian,  Ignorant,  and  noninformed  Intervals  retains  an  order. 
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The  effect  of  changing  x  for  fixed  n  Is  seen  to  be  a  change  In  the  order 
of  the  Fisherian,  Ignorant,  and  nonlnformed  Intervals,  The  Fisherian  Interval 
seems  to  be  the  widest.  For  x  near  n/2,  the  Ignorant  Interval  seems  to  be 
narrower  than  the  nonlnformed  Interval.  For  x  near  n  however,  the  nonlnformed 
Interval  seems  to  be  the  narrowest  of  the  three. 

Upper  One-Sided  Intervals; 

Figures  16  through  31  show  distributions  and  upper  one-sided  9(W 
confidence  Intervals  on  reliability.  The  lower  confidence  limit  appears  lower 
for  the  Fisherian  analysis  than  for  the  Bayesian  analyses.  The  Bayesian 
Ignorant  and  nonlnformed  priors  seem  to  lead  to  two  sets  of  results.  The 
lower  confidence  limit  appears  lower  for  the  nonlnformed  when  x  Is  near  n/2 
but  higher  for  the  nonlnformed  when  x  Is  near  n. 

The  symmetry  of  the  Beta  functions  makes  the  lower  confidence  limits  for 
x  near  0  such  that  the  Fisherian  Is  lowest,  the  nonlnformed  Bayesian  Is  next 
lowest,  and  the  Ignorant  Bayesian  Is  the  highest  of  the  three.  This  Is  shown 
In  Figures  25  though  28.  These  figures  and  Figures  29  through  31  also  show 
that  large  n  leads  to  fairly  close  agreement  between  the  three  methods. 

CONCLUSION 

The  three  methods  are  all  on  sound  theoretical  ground  but  give  different 
results.  No  single  method  provides  most  logical  confidence  Intervals.  The 
choice  between  methods  has  to  be  based  on  goals  and  philosophy.  Since  the 
Fisherian  method  leads  to  the  widest  confidence  Intervals,  It  Is  the  most 
conservative  approach.  Since  proponents  of  the  Bayesian  method  prefer  priors 
which  contain  more  Information  than  the  Ignorant  or  nonlnformed  prior,  the 
Bayesian  method  (without  a  prior  based  on  previous  tests/calculatlons)  does 
not  meet  all  the  goals  of  analysts  with  a  Bayesian  philosophy.  Thus  the 
Fisherian  method  seems  to  be  a  good,  conservative  method  for  the  Initial 
analysis.  This  initial  analysis  can  provide  a  prior  for  a  future  Bayesian 
analysis  of  addition  data  from  a  future  test. 
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Figure  16 
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Environmental  Sampling:  A  Casa  Study 


Dannis  L.  Brandon 
US  Army  Engineer 
Watervaya  Experiment  Station 
Vicksburg,  Mississippi  39180 


Abstract  Sampling  strategies  have  been  developed  to  accomplish 
various  environmental  objectives.  The  objectives  may  be:  (1)  to 
estimate  the  average  of  characteristics  in  a  population)  (2)  to  estimate 
Che  variability  of  characteristics  of  interest  in  a  population)  (3)  to 
decide  if  characteristics  of  interest  in  a  population  meet  certain 
standards  or  criteria)  (4)  to  identify  the  source(s)  which  caused 
characteristics  In  a  population  to  exceed  standards.  A  study  designed  to 
achieve  objectives  3  and  4  will  be  presented.  Modifications  and 
alternate  approaches  will  also  be  discussed. 

Background  Navigable  waterways  of  the  United  States  have  and  will 
continue  to  play  a  vital  role  in  the  nation's  development.  The  Corps,  in 
fulfilling  its  mission  to  maintain,  improve,  and  extend  these  waterways, 
is  responsible  for  the  dredging  and  disposal  of  large  volumes  of  sediment 
each  year.  Nationwide,  the  Corps  dredges  about  230  million  cubic  yards 
in  maintenance  vork  and  about  70  million  cubic  yards  in  new  dredging 
annually  at  a  cost  of  about  $430  million.  In  accomplishing  its  national 
dredging  and  regulatory  missions,  the  Corps  has  conducted  extensive 
research  and  development  in  the  field  of  dredged  material  management. 
Federal  expenditures  on  dredged  material  research,  monitoring,  and 
management  activities  have  cumulatively  exceeded  $100  million. 

Techniques  developed  to  evaluate  contaminant  mobility  in  dredged  material 
can  be  applied  to  other  contaminated  areas.  Accordingly,  the  plant  and 
animal  bioassays  are  two  techniques  developed  to  assess  the  environmental 
impact  of  dredged  material  in  vetland  and  upland  disposal  environments. 
These  bioassays,  surface  soil  samples,  groundvater  samples,  and 
additional  plant  tissues  wore  used  to  evaluate  a  contaminated  site  in 
western  California. 

The  case  study  site  is  approximately  200  acres  with  both  upland  and 
wetland  areas  (see  Figure  1).  This  site  was  known  to  have  very  high 
concentrations  of  metals  in  surface  soils.  Major  pathways  for 
contaminant  mobility  are  the  meandering  stream  which  flows  north  and  the 
drainage  ditches.  Also,  tidal  inundation  affects  a  substantial  portion 
of  this  site. 

The  objectives  of  the  study  were  to:  (1)  define  the  extent  of  the 
hazardous  substance  contamination  on  the  site)  (2)  identify  the  sources 
of  the  hazardous  substances  detected  on  the  property)  (3)  evaluate  the 
extent  of  migration  of  the  hazardous  substances  on  the  property)  (4) 
assess  the  bioavailability,  mobility,  and  toxicity  of  the  hazardous 
substances  detected  on  the  property)  (3)  evaluate  the  condition  of  the 
wetland  and  upland  habitats  on  the  property.  This  paper  focuses  on  the 
use  of  soil  samples  to  achieve  objectives  1  thru  4. 
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SAMPLING  PLAN  The  sampling  plan  was  formulated  based  on  previous 
soil  and  vater  data,  historical  information,  and  the  potential  pathways 
for  contaminant  mobility.  The  sampling  locations  are  shown  in  Figure  1. 
Three  samples  were  collected  at  some  locations  and  one  sample  was 
collected  at  the  other  locations.  The  triplicate  samples  vere  used  in 
statistical  comparisons.  This  sampling  plan  reduced  the  cost  of  the 
investigation  by  allowing  a  selected  number  of  sample  locations  to  be 
tested  extensively  while  other  sample  locations  roceivad  one-third  the 
cost  and  effort.  A  total  of  178  samples  vere  collected  and  analyzed  for 
As,  Cd,  Cu,  Fb,  Ni,  Se,  and  Zn. 

There  is  an  analogy  between  the  strategy  used  here  and  the  disposal 
philosophy  of  many  Corps  elements.  Most  dredging  and  disposal  decisions 
are  made  at  the  local  level  on  a  case  by  case  basis.  Often,  the 
environmental  objective  is  to  prevent  further  degradation  of  the  disposal 
area.  Therefore,  samples  are  collected  at  the  dredge  site  and  disposal 
site.  A  statistical  evaluation  performed  on  the  chemical  analysis  of  the 
samples  becomes  the  basis  for  determining  whether  degradation  will  occur. 
In  this  study,  samples  were  collected  at  the  remote  reference  area  and  an 
area  of  degradation  (i.e.  contamination).  Ten  triplicate  samples  were 
collected  in  the  remote  reference  area.  Twenty-eight  triplicate  samples 
vere  collected  in  the  area  of  contamination.  Locations  having  a  mean 
concentration  of  metals  in  soil,  plants,  or  animals  statistically  greater 
than  similar  data  from  all  remote  reference  locations  vere  declared 
contaminated.  These  concentrations  provide  a  judgemental  basis  for 
classifying  the  64  single  sample  locations. 

Three  sources  of  contamination  vere  identified  from  historical 
information.  One  additional  source  vas  indicated  by  the  soil  analysis 
and  later  verified  with  historical  information.  Sources  were  thought  to 
be  areas  with  several  high  metal  concentrations  in  a  vicinity  and  a 
gradual  decrease  in  metal  concentrations  as  one  moves  avay  from  this 
area.  The  sources  found  in  this  study  appeared  to  have  released  metal  in 
two  different  forms.  One  method  vas  to  bury  or  discharge  contaminants 
associated  with  a  solid  material  in  an  area.  The  other  source  discharged 
highly  contaminated  liquids  into  a  stream.  Identifying  sources  vas 
further  complicated  by  the  fact  that  some  of  the  discharges  vere 
intermittent  and  possibly  hadn't  occurred  in  several  years.  This  study 
vas  successful  in  identifying  sources  which  discharged  contaminants 
associated  with  solids.  Identifying  the  source  of  liquid  discharges  vas 
more  difficult  due  to  seasonal  fluctuation  of  the  stream. 

The  soil  analysis  vas  partially  successful  in  achieving  objectives  1 
thru  4.  The  extent  of  contamination  from  known  sources  vas  established 
and  locations  requiring  further  investigation  vere  identified.  This  plan 
has  been  augumented  with  additional  sampling.  These  samples  further 
delineated  th<.  extent  of  contamination  horizontally  across  the  site  and 
vertically  down  the  soil  profile.  As  a  result  of  this  study,  26.5  acres 
vere  declared  contaminated. 
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A  Generalized  Gumbel  Distribution 


Siegfried  H.  Lehnlgk 
Research  Directorate 

Research,  Development,  and  Engineering  Center 
U.S.  Army  Missile  Command 
Redstone  Arsenal,  AL  35398-5248 

A  generalized  Gumbel  (extreme  value  type  I)  distribution  class  Is 
Introduced.  In  addition  to  the  usual  shift  and  scale  parameters  this  new 
distribution  contains  an  arbitrary  positive  shape  parameter.  The  classical 
Gumbel  distribution  results  as  special  case  for  shape  equal  to  unity. 
Microcomputer-based  algorithms  for  estimation  of  the  parameters  are  present¬ 
ed.  They  are  based  on  the  moment  equations  and  on  the  logarithmic  likelihood 
function  associated  with  the  distribution  density.  A  program  diskette  for 
microcomputer  use  will  be  made  available  upon  request.  A  combined  paper  by 
this  author  and  Charles  E.  Hall,  Jr.,  will  be  published  elsewhere. 
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A  Generalization  of  the  Eulerian  Numbers  with  a 
Probabilistic  Application 


Bernard  Harris 

University  of  Wisconsin,  Madison 
C.  J.  Park 

San  Diego  State  University 


1  Introduction  and  Historical  Summary 


In  this  paper  we  study  a  generalization  of  the  Eulerian  numbers  and  a  class  of  polynomials  related 
to  them.  An  interesting  applicadon  to  probability  theory  is  given  in  Section  3.  There  we  use  these 
extended  Eulerian  numbers  to  construct  an  uncountably  infinite  family  of  latdce  random  variables 
whose  first  n  moments  concide  with  the  first  n  moments  of  the  sum  of  n  +  1  uniform  random 
variables.  A  number  of  combinatorial  identities  are  also  deduced. 

The  Eulerian  numbers  are  defined  by 


They  satisfy  the  recursion 


n+  i 
v 


()  “  y)"J  *  0, 1,2, •  •  > i it}  ft1*  0,1,2 . 


0) 


Ay  •  j  A*. »,/+(«-;'+ 


(2) 


and  the  Worpitzky  [25]  relation 


Also. 

Ay  ■  A  *,*-/♦!  }  (4) 

£A,y»nl. 

/■i 

In  addition,  they  possess  a  number  of  combinatorial  interpretations  which  are  dcecribed  below. 

Let  Xa  -  {1,2 . n)  and  let  P,(  Jk)  be  the  number  of  permutations  of  the  elements  of  X » 

having  exactly  k  increases  between  adjacsnt  elements,  the  first  element  always  being  counted  as 
an  increase. 

For  n  ■  4 ,  the  24  permutations  and  the  number  of  increases  are  given  In  Tbble  1.1. 
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Table  1.1 


pennutation  Number  of  tocmnwa 


1 

1234 

4 

2 

1243 

3 

3 

1324 

3 

4 

1342 

3 

5 

1423 

3 

6 

1432 

•  2 

7 

2134 

3 

8 

2143 

2 

9 

2314 

3 

10 

2341 

3 

11 

2413 

3 

12 

2431 

2 

13 

3124 

3 

14 

3142 

2 

15 

3214 

2 

16 

3241 

2 

17 

3412 

3 

18 

3421 

2 

19 

4123 

3 

20 

4132 

2 

21 

4213 

2 

22 

4231 

2 

23 

4312 

2 

24 

4321 

1 
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As  seen  from  the  tabulation,  P*  ( 1)  ■  1 ,  P4  ( 2)  ■  1 1 ,  P4  ( 3)  ■  1 1 ,  P4  ( 4)  ■  1 ,  which  concides 
with ■  1,2,3 r4. 

Let 

Mt)  -  XX*'-  (6) 

Then 

i ~ T^RT — >i  ■  giUWVi  J  i  1.  (7) 

The  above  relations  and  some  of  their  properties  can  be  found  in  [8];  the  polynomials  (6)  are 
also  discussed  in  L.  Carlitz  [4].  These  results  may  also  be  found  in  the  expository  paper  of  L. 
Corlitz  [3].  The  formulas  (1)  and  (2)  arc  also  given  in  L.  v.  Schrutka  [21]. 

D6sir6  Andr6  [  1  ]  established  that  is  the  number  of  permutations  of  { Xn  }  with  j  “elementary 
inversions”.  He  also  established  that  is  the  number  of  circular  permutations  of  }  with  ; 
“elementary  inversions”.  The  equivalence  of  these  two  results  with  the  enumeration  of  the  number 
of  increases  in  permutations  of  {X„}  can  be  trivially  established. 

G.  Frobenius  [15]  studied  the  polyomonials 

<A»(*)  -  ^lAnjd,  (8) 

i-l 

introduced  by  Euler,  and  established  many  of  their  properties.  In  particular,  relations  with  the 
Bernoulli  numbers  are  given  in  [15]. 

In  D.  P.  Roselle  [20],  the  enumeration  of  permutations  by  the  number  of  rises,  A ^ ,  is  related 
to  enumeration  by  the  number  of  successions,  that  is,  a  permutation  ir  of  {Xn}  has  a  succession  if 
7T(j)  -  »  +  1,»  »  l,2,...,n. 

Some  number  theoretic  properties  of  A^  are  given  in  L.  Carlitz  and  J.  Riordun  [7]  and  in  L. 
Corlitz  [5]. 

In  this  paper,  we  study  a  generalization  of  the  Eulerian  numbers.  A  generalization  in  a  dif¬ 
ferent  direction  was  given  by  E.  B,  Shanks  [22],  who  apparently  did  not  note  a  connection  of  his 
■coefficients  with  the  Eulerian  numbers.  L.  Carlitz  [2]  noted  the  relationship  of  Shank’s  results  to 
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the  Eulerian  numbers  and  obtained  representations  for  these  generalized  Eulerian  numbers  using 
.  results  due  to  N.  Nielsen  [17]. 

E  Poussin  [18]  considered  the  enumeration  of  the  number  of  inversions  of  permutations  of 
{Xn}  which  end  in  1  £  ;'  <£  n.  This  produces  a  decomposition  of  the  Eulerian  numbers.  She 
also  introduced  a  polyomial  generating  function  for  these  numbers.  The  sums  of  these  polynomials 
are  the  Euler-Frobenius  polynomials. 

Another  deomposition  of  the  Eulerian  numbers  with  a  combinatorial  interpretation  is  given  by 
J.F,  Dillon  and  D.P.  Roselle  [12]. 

J.  Riordan  [19]  lists  many  properties  of  the  Eulerian  numbers  in  Exercise  2,  page  38-39  and  de¬ 
scribes  the  combinatorial  interpretation  of  the  Eulerian  numbers  in  terms  of  triangular  permutations 
(which  is  equivalent  to  the  elementary  inversions  described  by  Andrd  [1]).  He  also  gives  a  brief 
table  of  the  Eulerian  numbers  on  page  215.  See  also  L.  Comtet  [10],  where  generating  functions  for 
the  Eulerian  numbers  are  given  and  the  Eulerian  numbers  are  obtained  by  enumerating  the  number 
of  permutations  with  a  specified  number  of  increases.  Many  properties  of  the  Eulerian  numbers  are 
given  as  well  as  their  historical  origins  in  terms  of  sums  of  powers. 

F.N.  David  and  D.E.  Barton  [11]  suggest  the  use  of  the  Eulerian  numbers  as  a  statistical  test  for 
the  randomness  of  a  sequence  of  observations  in  time,  employing  the  probability  distribution  given 
by 

Ptv  *^/n!t  ;  »  1,2 . n.  (9) 

The  generating  function  (7)  is  derived  and  employed  to  obtain  the  moments  and  cumulants 
of  the  distribution  (9).  In  particular,  David  and  Barton  show  that  the  factorial  moments  are  the 
generalized  Bernoulli  numbers.  However,  David  and  Barton  do  not  make  any  identification  of 
these  distributions  with  the  Eulerian  numbers. 

Using  probabilistic  arguments,  Carlitz,  Kurtz,  Scoville  and  Stackelberg  [6]  showed  that  the  Eu¬ 
lerian  numbers,  when  suitably  normalized,  have  an  asymptotically  standard  normal  distribution. 
This  was  accomplished  by  representing  the  distribution  as  the  distribution  of  a  sum  of  indepen- 
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dent  Bernoulli  random  variables.  S.  Tanny  [24]  demonstrated  the  asymptotic  normality  by  utilizing 
the  relationship  of  the  Eulerian  numbers  to  the  distribution  of  the  sum  of  independent  uniform  ran¬ 
dom  variables  and  applying  the  central  limit  theorem. 

L.  Takdcs  [23]  obtained  a  generalization  of  the  Eulerian  numbers  which  provide  the  solution 
to  a  specific  occupancy  problem.  Namely,  let  a  sequence  of  labelled  boxes  be  given,  the  first  box 
labelled  1,  the  second  box  2,  and  so  on.  At  trial  number  n  distribute  l  balls  randomly  in  the  first 
n  boxes  so  that  the  probability  that  each  ball  selects  a  specific  box  is  1  /n  and  the  selections  are 
stochastically  independent.  For  l  -  1 ,  the  probability  that  j  -  1  boxes  are  empty  after  trial  number 
n  is  A^/nl ■  1 , 2 , . . . ,  n.  Takdcs’  paper  contains  many  references  and  describes  additional 
combinatorial  problems  whose  solution  is  related  to  the  Eulerian  numbers. 

Finally,  L.  Toscano  [25]  obtained  formulas  expressing  the  Eulerian  numbers  in  terms  of  Stirling 
numbers  of  the  second  kind. 


2  Generalized  Eulerian  Numbers 


We  now  introduce  a  generalization  of  the  Eulerian  numbers  and  investigate  its  properties. 
Let  S  be  an  arbritrary  real  number  and  let 


4„(6)  -  E 

v-0 


n+  1  N 
v  / 


(-1)"(6  +  j  -  v)\j  “0,l,...,n;n«0,l,2,.... 


(10) 


These  polynomials  are  mentioned  in  L.  Carlitz,  D.P.  Roselle  and  R.A.  Scoville  [8],  As  noted 
there,  Aw  (  0)  are  the  Eulerian  numbers.  These  polynomials  are  also  used  by  P.S.  Dwyer  [13]  to 
calculate  sample  factorial  moments.  Dwyer  does  not  relate  these  to  the  Eulerian  numbers. 

We  begin  our  analysis  with  the  following  theorem: 

Theorem  1.  Let  n  and  k  be  non-negative  integers  and  let  S  be  any  real  number.  Then 
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Ml*] 


(«) 


(«  -  ^E(« +/>'*’  £  [  °+ 1  |  <-«•<«+/-  «)’ 

y-o  v-o  y  v  / 


»«  y.o 

is  independent  of  6  for  k  ■  0 , 1 , . . . ,  n. 

Proof.  The  following  identity  (see  N.  Nielsen,  [17],  page  28)  will  be  utilized  in  the  proof. 


(11) 


(® 


-«■-£(  *+;'1)£c-i)*(n+1  )(«+/-•>*. 

y-o  ^  n  J  ««o  y  v  j 


(12) 


Let  A  and  E  be  the  operators  defined  by 

A(/(a))  -  /(z+  1)  -f{x) 
and 

£(/<*))«/(*+!)• 
Then,  it  can  be  shown  that  (C.  Jordan,  [16]) 


r  It 

v  i  i 


««0  \  V 

In  particular,  for  r  ■  0 , 1 , . . . ,  n, 

Af<5  +  »<">  -£)<-!) 


(13) 


1  *  f  1  (6  +  J  +  r  —  0(n) 
i-o  V  l 


(14) 


-n<r)(6  +  /)("“r), 

the  last  equality  follows  from  elementary  properties  of  the  function  (6  +  jYn>  (C.  Jordan,  [16],  p. 
51).  Thus,  for  r  -  0 , 1 . .  from  (12)  we  have 


(6 +  ;)<*•">«  E(-D‘|  ’  |  (6  +  ;  +  r-0(B)/n(r). 
i-o  V  l 


(15) 
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Hence, 


-  3- £<«+»<*>£  f n+ 1  ]  (-1  )*(«+/- 

™  /*0  v-0  ^  v  J 


V)’ 


/-w  v«u  \  v  J 

4EE(-i)‘fn-*)({+/+n-*-i)<^fB+1 

nI  >0  M  \  l  j  ^  \  V 


Thus,  it  follows  that 


(-1  V(6  +  j-v)*/n'*-h>, 

(16) 


f v—h 

M[*](n)(6)  -E(-l) 

M 


n—  k  S 

\  ‘  i 

Setting  x  »  5  +  n-  fc  -  /  +  1  in  (12)  we  get 


E(s+i+"-k-‘)i(-.) 

>0  \  n  J  v>o 


)  h-»v 

f  n+  1  N 

J  v«0 

l  v  / 

(S+j-v)n/n{riFmk) , 
(17) 


and  hence 


,  l  1  1  \ n  A/ 6+n-fc-l  +  ;  \  >  /  n+ 1  \ 

£(-1)'  <6  +  /- 

/■°  \  n  J  w-o  \  v  / 


«)' 


tir'k 


( 


\ 


n~k  j  (n-k-l+  1  )n/n(*-w 


(18) 


W^-EM)1 

(-0 

and  is  independent  of  6. 

In  particular ,/i[nl(n)(6)  -  1,  A*i„-iSn) ( 5)  -  (2n-  l)/n  and  ^^(S) 

=  (3n-2w>l  +  l)/n(n  -  1),  ^,{<(0)  - 

A  brief  table  of  M[*|(n>  (6)  for  k  ■  0 , 1 , 2 , 3  and  n  ■  0 , 1 , 2 , 3  is  given  in  the  Appendix  to  this 
paper. 

The  Nielsen  identity  (12)  seems  to  have  been  discovered  in  a  somewhat  less  general  context  by 
Paul  S.  Dwyer  [13],  who  employed  it  to  calculate  factorial  moments  by  means  of  cumulative  sums! 
see  also  Ch.  A.  Charalam bides  [9],  who  in  addition  to  discussing  Dwyer’s  work  also  showed  thut 
these  generalized  Eulerian  are  related  to  enumeration  of  compositions  of  integers. 
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The  following  corollary  will  be  subsequently  employed. 
Corollary.  Let  n  and  k  be  non-negative  integers  with  k  <,71  Then 


pi*«)  -  -j- £<»+/)*£  ( n+ 1  I  (-!)•(«+/-»)* 

711  Jl.0  u-0  \  V 


(19) 


is  independent  of  5. 
Proof.  We  can  write 


r«0 

where  /3r*  are  the  Stirling  numbers  of  the  second  kind.  Since  the  coefficients  do  not  depend 
6,  substituting  (20)  into  (19)  a,nd  interchanging  the  order  of  summation,  we  get 


(20) 
on 


^<«-E/W°c  o. 

r»0 


(21) 


which  is  independent  of  6. 

Prior  to  demonstrating  that  the  independence  of  6,  noted  in  Theorem  1  and  its  corollary  „ 
be  extended  to  k  n  +  1 ,  we  will  need  to  calculate  the  derivative  of  ( 6) .  Thus,  we  have: 
Theorem  2.  Let  n  and  k  be  non-negative  integers  and  let  £  be  any  real  number.  Then 


can  not 


*  U"-1’ (». 


(22) 


Proof.  Since 


£(«+/>*£( n+1 

711  0  vaO  y  V  ) 


(_1  y(6  +  j~v)\ 


fn+1) 

™  /- 0  v»0  y  V  J 


(-1  )•(«  +  > -v)* 


\ 


J 
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Comparing  the  first  term  with  (19)  and  employing  the  Pascal  triangle  identity  on  the  second 
term  we  get 


do  ( TV—  1)1  ynO  \  vi  \  tl  -  1  J 


Further, 


(tl-  1)!  jmQ  v-Q  \  V  I 


+  urr>rg(4+^^(")(--lw+^^‘ 


From  (13) 


An(6"-1)  -£<~l)w(  n  (S  +  n-v)n~‘  -0, 

y»0  V  j 


since  it  is  the  nth  difference  of  a  polynomial  of  degree  n  -  1 .  The  second  term  on  the  right  hand 
side  of  (24)  is  /^"*l)($). 

In  addition, 

7-^  £<«+»*£  (  n  )<-iw+>-.>-‘ 

(n-  I)!  /To  vTo  \  v-  1  ) 

'  (TTT)T^(5+;',‘S( " )  D-1 

ln  1  '■»  »°W 

■7rnrr^(*+fl*£f "  ] 

171  ;«il  v- 0  \  V  / 
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“  jm 1  L  ««0  V  V  J 


(n 


1  *  (  k\ A 


^2: 

1 


M 


7Z - iTTZw  I  I  l  + 

(n~  1)!  l  r  )  fi{  v»o\v 


(-l)B+,((5-l +  /)-<*)  *’1 


7 — yrt  ( *  ]  E<«  +  ;>fE ( n  ]  (-1  >•*'<«+ /-10-1 

(n“  i)i  r<0  ^  r  y  /-o 


««o  V  v 


Sir 


(26) 


Thus,  by  (23),  (25)  and  (26)  we  have  shown  that 


r»o  V  f 


/ 


or 


Jk“l  / 


r«0 


Jb 

r 


(27) 


establishing  the  theorem. 
Corollary  1.  For  1  <,  k  <,  n,  then 


*-1 


(28) 


Proflf.  By  the  Corollary  to  Theorem  1,  ^  ( S)  is  independent  of  8  for  0  <,  k  <,  n  and  hence 
*  0  for  such  values  of  k> 

Corollary  2.  If  k  -  n  +  1 ,  then 


d6 


Cwt.il  -(n+  l)M»"“,)(fi)i 


(29) 


where  c^i,n  is  a  constant  (depending  on  n,  but  independent  of  6). 
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Proof.  From  the  Corollary  to  Theorem  1  and  Theorem  2,  ( n  +  l)^  ( $)  is  independent  of  6  and 

■  l  1  \ 

all  terms  in  with  the  exception  of  r  -  n  are  independent  of  8.  This  last 


term  is  (n+  l)/i^l)(fi). 

Corollary  2  can  be  extended  to  k  *  n+  2  and  so  forth,  but  the  expressions  obtained  become  more 
complicated  and  do  not  appear  to  be  particularly  useful.  However,  we  do  make  use  of  Corollary  2 
in  the  next  theorem, 

Theorem  3.  For  every  n,  ( $)  is  a  polynomial  of  degree  n  +  1  in  8  with  leading  coefficient 

<-Dn. 

Proof.  We  proceed  by  induction,  using  Corollary  2  to  Theorem  2. 


For  n»  0,/ii(6)  ■  8.  Then 


Performing  the  indicated  integration,  we  have 


where  d  is  an  unspecified  constant. 

Assume  that  the  conclusion  holds  for  n  ■  m.  Then 


dt&M 


Cm+2,m*l  (TO  *  i 


«<W2,m+l  -(m+2)(oo6m+‘  +  Eo^r1-^),  (30) 

>■  t 

where  a0  is  +1  or- 1.  Integrating,  we  get  ■  -oo$m+2  +  Pro+i(<5),  where  Pm+)(5)  is  a 

polynomial  of  degree  m  +  1 . 

A  table  of  8)  appears  in  the  Appendix  for  n  ■  5 ,  k  »  0 , 1 , . . . ,  10  and  selected  values  of 
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3  Applications  to  Probability  Theory 

Let  U\ ,  Ui , . . . ,  i  be  independent  random  variables  uniformly  distributed  on  ( 0 , 1) .  Let 
Swi  *  t/,.  The  distribution  of  Sn*  i  is  well-known  and  is  given  by  the  probability  density 

function 


•i  / 


/*..<*) -jj-E 


v-0 


n+  1 


V  v 


(for  example,  see  W.  Feller  [14].)  Where 


f 


(*-a)+ 


( —1)VC®  -  v)".0  <  x  <  n+  1, 


0,  x  -  o  <  0 
a:  -  a, 


(31) 


(32) 


Write 


SWi  *  [  SVi  ]  +  5, 

where  [5^i  ]  denotes  the  integer  part  of  S^\ .  Clearly  6  is  a  continuous  random  variable  and 

0  <;  8  <  1;  [  Sm- 1  ]  is  a  discrete  random  variable  with  carrier  sett  {0,1,2 . n}. 

The  conditional  distribution  of  given  that  the  fractional  part  of  SUi  is  £  is  given  by 

P{S»\  ■  x\x-  [a]  -  5}  -/s„,(;'  +  4)/£/*wi(>  +  5),  (33) 

>■  o 

where  +  6  ■  x,  j  ■  0 , 1 , . . . ,  n;  i.e.  j  ■  [  ®] . 

From  (31), 

ntl  f  n+  1  ^ 


1 


fsm i  O'  +  fi)  “  rr  2 

nl  v-0  V  V 


(-!)"(>  +  «-*)♦". 


/ 


But;  +  6  -  v  ^  0  is  equivalent  to  v  <,  j,  thus  we  get 


/*.,(/  +  »  (  "+  ‘  |  (-!)■(;  +  «-*)", 


(34) 


v-0  V  V 


which  is  Ay(6)/n\,  Also, 


E oiw(« 

;-o 
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and  thus  (34)  is  a  discrete  probability  distribution  with  carrier  set  {5, 1  +  8, . . . ,  n  +  6}. 

Let  Wnti.s  be  die  random  variable  whose  distribution  is  given  by  (34).  We  then  have  the  fol¬ 
lowing  theorem. 

Theorem  4.  The  moments  of  order  k  «  0 , 1 , . , . ,  n  of  i,«  coincide  with  the  corresponding 
moments  of  i ,  that  is, 

0,1 . njOS«<l.  (35) 

Em2£.  £{$*.,}-  £,(£{^,15})  -  E,(E{Wilf}}, 

However, 

£{"£,,,} -^-£0'+S>‘£[  B+1  |(-1)”(/  +  S-V)’, 

711  ;'.0  v-0  (  tl  J 

which  is  independent  of  6,  by  the  Corollary  to  Theorem  1. 

A  brief  table  of  PVn+  i  ,s  for  n  *  5  is  given  in  the  Appendix. 

Remark.  It  is  easy  to  see  that  the  marginal  distribution  of  6,  the  fractional  part  of  SU  i ,  is  uniform 
on  (0 , 1) .  An  elementary  proof  follows. 

n  rM* 

P{SZS’}-t2[  /«.,(*)<(* 

;*0 

fs„iV  +  u)du 

>■  o J0 

•  f.  ,(;  +  u))du; 

;*0 

but  i  (/  +  u)  -  1  for  every  0  <,  u  <  1 .  Hence 

P{$  £  5’}  -  f*  du  -  S'. 

Finally,  we  note  that  W*  i ,«  is  asymptotically  normally  distributed.  This  is  stated  in  the  following 
theorem. 
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(36) 


Theorem  5.  As  n  -■*  oo,  for  0  £  £  <  1 ,  the  distribution  of 

JSnr (*)) 

coverges  weakly  to  the  standard  normal  distribution. 

Further, 

.  JHe-iO^-^D/^ci  +  cxn-h).  (37) 

n!  yirn 

Proof.  Both  (36)  and  (37)  are  immediate  consequences  of  the  representation  of  as  the  con- 
didonal  distribution  of  the  sum  of  n+  1  independent  uniform  random  variables  on  [0 , 1)  given  the 
fractional  part  of  the  sum  and  the  central  limit  theorem. 
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Appendix 


This  Appendix  is  devoted  to  some  tables  illustrative  of  some  of  the  quantities  introduced  in  the 
body  of  the  paper. 


Table  A.l 


Table  of/i$(6),Jfe  =  0, 1 

,2,3;n»0,l,2,3 

A 

n 

0 

1 

2 

3 

0 

1 

6 

5(6-1) 

6(6-  1)(6 -  2) 

1 

1 

jW-1 

z"‘ 

1  +  6- 6s 

263  +  362  -  6  -  1  +  n 

2 

1 

ntL 

2 

LatiHM. 

12 

1  +  26  -26*  -  tatillfl. 

3 

1 

(feUtitnA 

12 

(itt.lKnrlitefl- 

24  • 
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Table  A. 2 


The  Distribution  of  n»  5,  6  *  .1,.4,.5,.9 


w. 

L±A 

iioi 

6-».5 

6  m  -9 

6 

8  x  10-* 

9  x  10“5 

3  x  10“4 

.005 

1  +  6 

.013 

.044 

.062 

.177 

2  +  6 

.260 

.396 

,438' 

.545 

3+6 

.545 

.476 

v*> 

oo 

.260 

4+6 

.177 

.083 

,062 

.013 

4+6 

,005 

6  x  10-4 

3  x  15  “4 

5  x  10" 

Note  the  symmetry  for  6  ■  .5  and  that  6  »  ,9  and  6  *  .  1  are  identical  when  the  column  for 
6  *  .9  is  read  going  up  and  6  *■  ,1  is  read  going  down  (the  entries  8  x  10  “*  and  5  x  10  “8  differ 
as  a  consequence  of  rounding  errors). 
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TflbkA3 


5 , ■  0, 1, ,  10; fi  ®  0, .1,  .3, .5,  .7, .9 


o> 

I 

O 

.1 

.3 

.5 

.7 

.9 

k  *  0 

1 

1 

1 

1 

1 

1 

1 

3 

3 

3 

3 

3 

3 

2 

9.5 

9.5 

9.5 

9.5 

9,5 

9.5 

3 

31,5 

31,5 

31,5 

31.5 

31.5 

31.5 

4 

108.7 

108.7 

108,7 

108,7 

108.7 

108.7 

5 

388.5 

388.5 

388.5 

388.5 

388.5 

388.5 

6 

1432.50 

1432.50 

1432.53 

1432 .55 

1432.53 

1432.50 

7 

5431.50 

5431.51 

5432.01 

5432 .48 

5432.31 

5431.69 

8 

2118.7 

21117.60 

21122.56 

21129.77 

21129.66 

21122.07 

9 

84010.5 

83989.19 

84020.48 

84096.88 

84116.67 

84049.80 

10 

341270.5 

341018,48 

341121.81 

341763.40 

342089.16 

341628.77 
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Using  an  Ordered  Categorial  Approach 
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ABSTRACT 


When  the  experimental  units  being  classified  are  sub-sampling  units  in  the  study,  an 
ordered  categorical  procedure  cannot  be  applied  directly.  Further,  the  count  data  ob¬ 
tained  which  is  routinely  analyzed  by  univariate  statistical  methods,  ignores  the  depen¬ 
dence  among  the  responses.  A  modification  of  the  method  developed  by  Nair  (1986,  1987) 
is  used  to  derive  the  scores  and  indices,  which  are  analyzed  by  nonparametric  AOV.  An 
example  from  teratogenicity  studies  is  used  to  illustrate  the  technique. 

Introduction 

This  problem  arises  from  the  consideration  of  studies  where  a  reproduction  Bafety  test 
must  be  performed  prior  to  the  use  of  drug,  chemical1  or  food  additive.  The  standard  pro¬ 
tocol  in  such  studies  requires  that  pregnant  female  subjects  (usually  rodents)  are  randomly 
assigned  to  one  of  four  treatment  groups.  The  appropriate  dosage  is  administered  shortly 
after  the  beginning  of  gestation.  When  the  animals  are  near  term,  they  are  sacrificed 
and  the  number  of  potential  offspring  are  counted.  Other  data  collected  are  the  number 
of  implantation,  early  and  late  fetal  deaths,  number  of  live  offspring  and  the  number  of 
fetuses  according  to  various  degrees  of  increasing  severity  of  malformation.  Also  data  on 
continuous  variables  such  as  fetal  weight  are  collected.  It  is  unclear  from  the  literature 
which  statistical  methods  are  appropriate  for  the  analysis  of  this  type  of  data. 

For  continuous  measurements  one  may  quickly  turn  to  the  analysis  of  variance.  For 
count  data  describing  the  number  of  fetuses  with  or  without  some  qualitative  outcome, 
other  methods  have  evolved.  A  per-fetus  analysis  using  total  of  early  death  and  total 
number  of  implantation  in  a  Fisher  exact-test  or  a  chi-  squared  test  of  independence  may 
be  performed,  but  this  appears  to  inflate  samples  sizes  and  ignores  the  dependence  of 
observations  within  litters.  A  review  of  per-fetus  analysis  is  given  by  Haseman  and  Hogan 
(1975)  who  conclude  the  per  letter  analysis  is  more  appropriate. 

All  but  one  of  the  proposed  methods  for  per-litter  analysis  consider  a  single  outcome. 
The  need  to  include  within  and  among-litter  variation  negates  the  use  of  simple  binomial 
or  Poisson  models  for  count  data.  In  the  methods  which  consider  several  single  responses, 
a  problem  of  family  error  rate  arises.  Since  the  tests  arc  not  independent,  the  nominal 
family  error  rate  cannot  be  exactly  determined,  The  multivariate  method  developed  by 
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Ryttman  (1976)  relies  on  the  assumption  of  normality  which  is  violated  in  the  case  of 
fetal  deaths.  This  lack  of  success,  however,  does  not  preclude  a  multivariate  approach.  In 
situations  where  ranking  the  categories  from  mild  to  severe  is  possible,  ordered  categorical 
models  may  be  applied  and  the  family  error  problem  may  be  eliminated. 

In  this  paper  we  obtain  a  scoring  system  for  various  outcomes  which  produces  a  severity 
index  for  each  litter.  This  index  is  sensitive  to  location  shifts.  The  modeling  which  follows 
will  be  based  on  this  index. 

The  3tudy  design  prohibits  the  straight-forward  application  of  ordered  categorical  pro¬ 
cedures  because  the  items  (fetuses)  are  not  independent,  Thus  a  scoring  procedure  allows 
consideration  of  the  effect  of  letter  size  on  severity  of  the  response,  as  a  whole,  in  the  litter, 
Here  the  sampling  unit  is  the  fetus  or  individual.  Three  observations  should  be  made;  i) 
results  are  different  per  litter  than  for  per  fetus,  ii)  per  litter  evaluates  the  proportion  of 
fetuses  affected  rather  than  the  numbers  of  affected  litters,  and  iii)  observed  treatment  - 
control  differences  is  less  significant  than  per-fetus  indicates  (via  simulation). 

Univariate  Analysis. 

The  simple  analysis  is  based  on  litter  as  the  experimental  unit.  This  analysis  is  carried 
out  using  binomial  and  poisson  models.  The  binomial  assumption  states  that  conditional 
on  litter  size  the  number  affected  is  binomial.  The  analysis  is  based  on  transformed  data, 
usually  the  arc-sine  of  the  observed  proportion.  The  poisson  model  does  not  account  for 
litter  size  as  it  assumes  the  mean  number  affected  is  the  same  for  all  dose  groups.  The 
analysis  again  used  a  transformation,  usually  the  square  root  of  the  observed  number. 
Neither  fits  the  data  very  well.  This  may  be  due  to  extra  binomial  or  extra  poisson 
variability,  as  the  case  may  be. 

More  sophisticated  models  are  reviewed  by  Haseman  and  Kupper  ( 1970 )  include:  weighted 
least  squares  based  on  proportion  and  unequal  sample  sizes.  This  approach  due  to  Cochran 
(1943)  requires  sample  sizes  which  are  too  large  for  this  application.  Others  include,  the 
normal-binomial  (Luning  et.  al.  1966),  beta-binomial  (Williams  1975),  negative  binomial 
(McCaughran  and  Arnold  1976),  correlated  -  binomial  Altham  (1978),  jackknife  Gin  den 
(1979).  Several  nonparametric  procedures  have  been  tried,  namely  the  Mann- Whitney  U, 
the  Kruskal- Wallis  and  the  Jouckheere/  Terspstra,  Some  attempts  at  multivariate  analy¬ 
sis  have  been  tried  by  Ryttman  (1976),  log-linear  models  by  Haberman  and  others  (1074) 
and  generalized  linear  models  by  McCullagh  (1980),  All  of  the  latter  techniques  have 
distributional  assumptions. 

Since  some  of  the  ordered  categorical  procedures  develop  or  accept  scores  for  the  cat¬ 
egories,  this  approach  was  pursued.  Scores  induce  relative  spacing  among  the  categories. 
Thus,  a  mean  score  may  be  obtained  for  each  litter.  This  implies  analysis  by  litter  as  a 
sampling  unit.  We  note  that  CATMOD  in  SAS  allows  for  scoring,  but  the  scores  must  be 
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user  specified. 

Ipsen  (1955)  suggested  a  scoring  for  Bioassay,  Instead  of  estimating  an  LDr, q  or  ED 50 
based  on  number  of  survivors  after  x  days,  he  ordered  the  data  into  categories  with  the 
continuum  represented  by  time  (days).  The  scores  proposed  are  such  that  the  variance  of 
the  linear  regression  of  mean  scores  on  the  log  dose  is  maximized  with  respect  to  the  total 
variance.  An  adjustment  is  made  if  the  scores  do  not  reflect  the  ordering  of  the  categories. 

Bradley  et.al.  (1962)  scores  by  maximizing  the  treatment  sum  of  squares  after  scaling 
relative  to  the  error  3urn  of  squares.  This  is  an  iterative  procedure  which  does  not  require 
the  assumption  of  linearity. 

Using  no  distributional  assumption,  Nair  (1986,  1987)  suggested  some  techniques  for 
analyzing  ordered  categorical  data  in  the  field  of  quality  control.  He  showed  the  Taguchi 
statistic  for  2x2  tables,  can  be  orthogonally  decomposed  into  K  -  1  components  where 
K  is  the  number  of  categories.  In  the  two  sample  case  he  showed  the  first  components 
is  equivalent  to  Wilcoxon’s  rank  test  on  grouped  data,  Thus,  this  components  would  be 
sensitive  to  shifts  in  the  multinomial  model.  Further,  the  second  components  corresponds 
to  Mood’s  rank  test  for  grouped  data,  thus  is  sensitive  to  scale  changes  in  the  2  x  K  model, 

In  the  non  equiprobable  case  the  correspondence  does  not  apply  though  the  interpre¬ 
tation  still  holds.  This  result  has  been  verified  using  a  comparison  density  approach  for 
the  two  sample  problem  by  Eubank,  LaRiccia  and  Rosenstein  (1987). 

The  decomposition  of  Taguchi’s  accumulation  chi-squared  (1966,  1974)  requires  the 
solution  of  an  eigenvector  problem.  Nair  (1986,  1987)  provides  the  method  for  deriving  two 
sets  of  scores.  These  yield  statistics  that  are  approximately  equal  to  those  obtained  from 
the  orthogonal  decomposition,  but  do  not  require  a  rigorous  solution.  The  approximate 
and  exact  statistics  have  comparable  power, 

When  applied  t.o  2  x  Iv  tables,  the  first  set  of  Nair’s  scores  is  sensitive  to  shifts  in  locution 
of  the  underlying  random  variable.  It  is  reasonable  to  suggest,  when  applied  to  litters, 
these  scores  yield  a  continuous  index  useful  for  detecting  shift.  In  teratogenicity  studies 
the  location  shifts  of  interest  would  be  those  that  indicate  a  significant  close-response. 

Nnir's  Method 

As  already  mentioned,  the  first  and  second  components  of  the  orthogonal  decomposition 
correspond  to  the  Wilcoxon  and  Mood  rank  test,  respectively. 

Wilcoxon  rests, 


ffu  :  G(. v)  =  Fix) 

Hi  :  G(x)  =  F(x  -  6) 
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where  F,  G  are  two  distribution  functions. 


Mood  tests, 


Ho  :  G(x)  s  F(x) 

^  :  G{ x)  =  F(x/9) 

where  9  is  a  constant. 


For  more  than  two  treatment  groups  the  first  component  corresponds  to  the  Kruskal 
-  Wallis  statistic  for  grouped  data,  and  the  second  to  the  generalized  Mood  statistic.  In 
the  general  case  (except  equiprobable)  case  the  equalities  are  no  longer  exact,  but  the  first 
two  components  have  good  power  for  detecting  location  and  scale  shifts  respectively.  The 
focus  of  this  work  is  on  location  shifts. 

Natation 


Observed  frequency  for  the  (i,k)th  cell  =  i  =  1,2 

j  —  1,2,  ...A' 

Column  total.  =  Ck  =  Y\k  +  Yii< 

Row  total.  =  Ri  =  Ri  =  T.kssi  Yik 
Cumulative  row  frequencies  Ztk  ~  Y.k}^\  Y*j 
Cumulative  column  totals.  A,  -  ZU  'i 
The  row  proportions  =  r\  =a  H;/JV 
The  column  proportions  =  Ck  =  Cy/N 

Cumulative  column  proportion  up  to  and  including  column  k  d *.  =  D^/N, 


Vector  conventions  used. 


bold  lower  case  letter  =  a  vector 
bold  upper  case  letter  =  a  matrix 
transpose  =  ( 

a  vector  raised  to  a  power  implies  each  element  is  raised  to  a  power 
(this  is  non  standard) 

Multinomial  model.  2  X  K  case 


Two  random  samples  of  size  Ri  i  =  1,2  are  drawn  from  two 
multinomial  populations, 

For  each  population,  the  probabilities  of  the  K  outcomes  are  given  by 
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(* 


Pik  k  -  1,  2,  ...  K. 

The  cumulative  probabilities  for  population  i  are  given  by 
Tik  =  Pit 

If  the  K  categories  are  assumed  to  be  ordered  the  hypothesis  is 

H0  :  Tv,  -  Tik  k  =  1,  2,  ...  K 
H\  :  (rik  —  Tik)  <  0  for  all  K 
(strict  inequality  for  at  least  one  k). 

Alternative  statistics  to  Pearson’s  x2  . 

K  1 

Taguchi's  statistic  TE  =  £  [4(1  -  4)]“1{S  Ri(Z{k/Ri  -  4)} 

Awl  i=t 


If  xlp  is  the  Pearson  x2  statistic  from  a  2  x  2  table  where  column  1  contains  the 
cumulative  frequencies  of  categories  1  through  k  and  column  2  contains  the  cumulative 
frequencies  of  categories  (k+1)  through  K.  Then, 

A'-i 

Te='£x\p 

Aol 

where  Te  is  a  “ccs  type  statistic”. 

Te  assigns  weight  tu*  =  [4(1  —  4)]“l  to  the  kth  term  in  the  sum  which  is  equal  for  each 
k  under  H0, 

K~  1  2 

Nair's  Statistic  T  =  £  Wk  Ri(zik/R<  -  4)2 

Asl  i— 1 


The  statistics  in  the  class  are  obtained  by  the  choice  of  the  set  {u>*}  where  u>*  >  0 
for  k  =  1,  2,  ...  A'  -  1.  The  decomposition  is  carried  out  conditionally  on  the  marginal 
proportions.  For  yi,  Wk  k  =  1,  2,  ...  K  —  1,  W  is  a  diagonal  matrix.  Using  the  cl  we  form 
the  (A'  —  1)  x  K  matrix  A  by; 


f  1  -4 

-4 . 

-4 

\ 

1  -  4 

l  -  4 

— — 4 

\  i  -  4-i 

i  —  4-i— 

•  i  ~  4--i  - '4- 

/ 
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Thus  T  is  given  by 


T  =  y^W AyxjNr^ri  . 
To  express  T  as  a  sum  of  squares  in  y\  we  need  to  express 


A'WA 


as  a  product  of  a  diagonal  matrix,  Q  and  its  transpose.  Let  A  be  diagonal  of  order  K 
formed  by  the  colum  proportions  {c*}  and  let  F  be  the  diagonal  matrix  of  order  (k  —  1) 
containing  the  eigenvalues  of  A*W AA  then  the  decomposition  yields 

A'WA  =  QVQ 1 

where  Q  contained  the  eigenvectors  of  A'WAA  such  that 


V  =  [i  I  9] 

satisfies 


Z?  A<?  =  I 


substituting  QTQt  into  T  above  with 


u  =  Q\Nr 


yields 


K- 1 

=  E  rtf 

j=  1 


where  the  r/s  are  elements  of  the  vector  of  eigenvalues,  r,  and  Uf  s  are  elements  of  it. 
Under  Hq  the  distribution  of  ij\  conditional  on  row  and  column  proportions  is  multiple 
hypergeometric  with 
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E(Ylk)  =  Nrxck 

Cov(Ylk ,  Yu)  =  N(  1  -  l/JV)“1r1ra(l  -  cfc)  *  =  1 

=  —  JV(1  —  1/N)r\r2(ckci)  k  ^  1 

or 

■£(2/1)  =  iVriAl 

cov(yi  =  JV(1  -  l/iV)-1r1r2A[(J  -  1)  •  l‘A] 


1  is  a  A'  x  1  vector  of  ones.  It  follows  that 

E(u)  =  NrxiQ*^  1)  /  s/Nr^r?  =  0 

cov(u)  m  (1  -  l/iV)-lQ‘A[/  -  1  1*A]Q  *  (1  -  1/iV)-1  J 


implies  the  Uj'a  are  uncorrelated  with  zero  means. 

Under  H0  it  can  be  shown  that  the  limiting  distribution  of  yx  converges  to  the  multi¬ 
variate  normal  distribution  as  N  goes  to  infinity.  Thus 

K- 1  K- 1 

i-x  jw 1 

a  weighted  sura  of  independent  x!  random  variables,  each  with  1  df 

The  approximate  solution  by  Nair  proposed  two  sets  of  statistics  which  have  properties 
the  same  as  those  obtained  for  the  equiprobable  case  (i.e.  ck  =  1  fk).  That  is  the  first 
component  of  T#  or  UeiX  is  equivalent  to  the  Wilcoxon  test  on  the  2  x  K  table  and  Ue,2> 
the  second  component,  is  equivalent  to  Mood’s 

M  =  £  [k  -  (I<  +  l)/2 ]2Ylfc. 

k=\ 

They  do  not  require  the  solution  of  the  eigenvalue  problem  as  orthogonal  decomposi¬ 
tion  is  not  necessary.  The  first  component,  all  observations  in  the  category  are  assigned  a 
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score  proportional  to  the  midpoint  of  the  category.  The  second  component  the  scores  are 
quadratic  in  the  midrank.  Additionally,  each  set  of  scores  is  adjusted  to  satisfy  orthogo¬ 
nality. 

To  calculate  the  scores,  let  c  be  of  length  K  with  elements  being  the  column  proportions. 
Form 

/  .5  0. 

1  .5 

B  =  : 

1  1 
V  1  1 

Let  r  =s  Be  and  r*  =  r  -  .5(1).  Note  the  r’s  are  Bross’s  ridits.  The  first  set  of  scores  is 
obtained  from 

1  -  r*  /  s/cV2 

where  r  *2  is  a  vector  of  squares  of  elements  of  r  *  .  The  second  set  of  scores  is  obtained  in 
two  steps.  First  let 

e  =  l‘[l  -  (c‘l3)l]  -  1. 

Then 

a  =  e/c‘ea. 

The  approximate  statistics  for  the  2  x  K  table  are 

V?  =  L\/Ri  +  L22/R7 

where 

Li  =  l*y,-  1,  2 

and 

V  =  S^/Rx  +  S22/R2 

where 

Sj  =  sl  i  =  1,  2 
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which  are  comparable  in  magnitude  and  consequently  in  power  to  Ux  and  U2  respectively. 
We  now  apply  the  method: 

Conduct  of  the  study  and  data: 

PROTOCOL 

1.  Sprague-Dawley  rat  study 

2.  Herbicide:  nitrofen  (2,  4-dichloro-4  nitrodipheynl  ether) 

3.  Test  compound  administered  during  organogenesis 

4.  Sacrifice  prior  to  parturition  and  cesarean-sectioned 

5.  Record  litter  a„.d  fetal  data 

6.  Administration  of  compound  follows  daily  dose  regimen 

7.  Treatment  groups;  control  and  three  dose  groups 

S.  Inseminated  females  randomly  assigned  to  4  groups  of  24  rats  each 

9.  Dose  levek.  6.25,  12.5,  25  mg/kg/day  body  weight  on  days  6-15  of  gestation. 

Controls:  gavage  solution  w/o  test  compound 


Live  fetuses  are  weighed,  sexed  and  examined  for  external  malformations.  They  are 
then  sacrificed  in  order  to  perform  the  skeletal  and  visceral  examination.  Recorded  are  the 
number  of  corpora  luta  on  each  ovary,  number  of  implantations,  number  of  fetuses,  and 
the  number  of  resportions  in  each  utrine  horn.  Table  I  displays  the  data  for  each  rodent 
and  close  level. 

The  following  definitions  are  employed  to  categorize  the  fetuses:  Dead  -  Early  or  late 
resorption  of  dead  at  c-section,  malformed-gross  visceral  or  skeletal  variation,  growth  re¬ 
tarded  -  body  weight  more  than  two  standard  deviation  from  the  mean  for  the  given  sex 
or  by  a  range  test.  Normal  -  absence  of  any  of  the  previous  outcomes.  Table’s  II  and 
III  summarize  the  results  by  number  and  percent  for  each  dose  by  category.  It  should  be 
noted  that  the  differing  number  of  letters  is  due  to  nonpregnant  females,  not  toxicity. 

The  final  columm  of  Table  I  is  the  calculated  severity  index.  This  index  is  calculated  by 
multiplying  the  score  for  the  category  by  the  number  of  fetuses  in  the  category,  summing 
and  dividing  by  the  number  of  implantations,  i.e., 


SI  =  n‘c/n'1. 


Details  of  the  calculations  of  a  severity  index  are  given  in  the  following  example. 


Consider  the  following  sample  data: 


Table  I 

Nitrofen  Data  -  Strague-Dawley  Flats 


Id 

Number  of 
Implantations 

Normal 

Growth 

Retared  Malformed 

Dead 

Severity 

Index 

SJTiMSia 

SHMM 

19 

1 

1 

0 

0 

0 

0.00000 

8 

4 

3 

0 

0 

1 

0.68423 

11 

5 

4 

1 

0 

0 

0.25139 

7 

8 

7 

0 

0 

1 

0.34212 

1 

12 

9 

0 

3 

0 

0.49145 

16 

14 

14 

0 

0 

0 

0,00000 

24 

14 

11 

0 

3 

0 

0.42124 

6 

15 

15 

0 

0 

0 

0.00000 

9 

15 

13 

0 

1 

1 

0.31352 

20 

15 

12 

0 

3 

0 

0.39316 

22 

15 

12 

1 

2 

0 

0.34590 

2 

16 

14 

1 

1 

0 

0.20142 

4 

18 

16 

0 

0 

0 

0.00000 

16 

16 

0 

0 

0 

0.00000 

12 

16 

14 

2 

0 

0 

0.15712 

17 

18 

16 

0 

0 

0 

0.00000 

23 

16 

15 

0 

0 

1 

0.12286 

3 

17 

11 

0 

6 

0 

0.69382 

5 

17 

10 

0 

6 

1 

0.85481 

13 

17  • 

13 

0 

3 

1 

0.50790 

15 

17 

13 

0 

2 

2 

0.55326 

21 

18 

13 

0 

4 

1 

0.58890 

Dose  Group  **  Low  (6.25  m 

g/kg/day  b.w. 

> . 

32 

1 

0 

0 

0 

1 

2.73692 

28 

12 

10 

0 

2 

0 

0.32763 

43 

12 

9 

0 

3 

0 

0.49145 

26 

14 

10 

0 

4 

0 

0.56166 

31 

14 

14 

0 

0 

0 

0.00000 

39 

14 

11 

0 

3 

0 

0.42124 

41 

14 

9 

0 

5 

0 

0.70207 

47 

14 

10 

0 

4 

0 

0.56166 

48 

14 

14 

0 

0 

0 

0.00000 

33 

15 

10 

0 

5 

0 

0.65527 

38 

15 

10 

0 

5 

0 

0.65527 

40 

15 

13 

0 

1 

1 

0.31352 

45 

15 

12 

0 

3 

0 

0.39316 

25 

16 

11 

0 

5 

0 

0,61432 

27 

16 

12 

0 

4 

0 

0.49145 

34 

16 

10 

0 

6 

0 

0.73718 

35 

16 

12 

0 

4 

0 

0.49145 

.37 

16 

9 

0 

7 

0 

0.86004 

44 

16 

12 

0 

4 

0 

0.49145 

46 

16 

11 

0 

5 

0 

0.61432 

36 

17 

7 

0 

8 

2 

1.24708 
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Table  I  (Cont’d.) 


Nitrofen  Data 

-  Strague-Dawley  Rats 

Number  of 

Growth 

Severity 

Id 

Implantations 

Normal 

Retared 

Malformed 

Dead 

Index 

EmBSBEEmKHa 

g/kg/day  b.w.) 

54 

2 

0 

0 

0 

2 

2.73692 

70 

3 

1 

0 

2 

0 

1.31054 

59 

4 

2 

0 

2 

0 

0.98290 

64 

8 

5 

0 

3 

0 

0.73718 

53 

11 

4 

1 

5 

1 

1.25663 

55 

13 

7 

0 

5 

1 

0.96661 

58 

14 

7 

0 

6 

1 

1.03798 

60 

14 

7 

0 

5 

2 

1.09306 

65 

14 

8 

0 

5 

1 

0.89757 

68 

14 

10 

0 

3 

1 

0.61674 

62 

15 

6 

0 

6 

3 

1.33371 

67 

15 

13 

0 

1 

1 

0.31352 

71 

15 

8 

0 

4 

3 

1.07160 

49 

16 

6 

0 

10 

0 

1.22863 

69 

16 

11 

0 

4 

1 

0.66251 

56 

18 

15 

0 

2 

1 

0.37047 

57 

18 

13 

0 

5 

0 

0.54606 

72 

18 

7 

0 

11 

0 

1.20133 

Dose  Group  =  High  (25,0'  mg/kg/day  b.w.) 


91 

2 

0 

0 

1 

1 

2.35136 

80 

7 

3 

0 

3 

1 

1.23348 

86 

8 

6 

1 

1 

0 

0.402S4 

73 

10 

1 

0 

9 

0 

1.76923 

77 

14 

3 

0 

11 

0 

1.54456 

78 

14 

3 

0 

11 

0 

1.54456 

79 

14 

2 

0 

12 

0 

1.68498 

83 

14 

9 

0 

5 

0 

0.70207 

93 

14 

1 

0 

12 

1 

1.88047 

76 

15 

0 

0 

15 

0 

1.96581 

84 

15 

6 

0 

D 

0 

1.17949 

92 

15 

*■» 

0 

8 

0 

1.04843 

74 

16 

4 

0 

11 

1 

1.52255 

87 

16 

6 

0 

10 

0 

1.22863 

94 

16 

8 

0 

8 

0 

0.98290 

95 

16 

6 

0 

10 

0 

1.22863 

96 

16 

4 

0 

10 

2 

1.57075 

89 

17 

6 

0 

11 

0 

1.27199 

90 

17 

0 

0 

11 

6 

2.23797 

75 

18 

6 

0 

12 

0 

1.31054 

81 

18 

6 

0 

12 

0 

1.31054 

S8 

19 

11 

0 

7 

1 

0.86829 

Table  II 


Number  of  Implantations 

Normal 

■rasm 

Control 

5 

8 

Low 

1 

5 

334 

Mid 

■ 

1 

79 

18 

228 

High 

98 

1 

199 

13 

311 

Table  III 


fti'isf 

Percent  oi:  Implantations 

Group 

IKBI 

Malformed 

Dead 

Control 

84.0 

1.7 

11.7 

2.7 

Low 

71.8 

0.3 

26.6 

1.5 

Mid 

57.0 

0.4 

34.6 

7.9 

High 

31.5 

0,3 

64.0 

4.2 

no 


Number  of  Implantations  (Fetuses) 

Normal 

Gr.Retarded 

Malformed 

Dead 

Totals 

Control 

252 

5 

35 

8 

300 

Low  Dose 

239 

1 

89 

5 

334 

Middle  Dose 

130 

1 

79 

18 

228 

High  Dose 

98 

1 

199 

13 

311 

Totals 

719 

8 

402 

44 

1173 

Calculate  the  column  proportions: 

ck  .61295823 

.00682012 

.34271100 

.03751066 

Calculate  Bross’s  ridits  (1958)  by  the  formula  r*  =  (c04-ci  4-. .  .  +  c^_i)  +  -5cfc  where  c0  =  0  : 
rk  .30647912  .61636829  ,79113385  .98124468 

Now,  let  rk  —  rk  -  .5  ; 

rj  -.19352088  .11636829  .29113385  .48124468 

Calculate  the  constant  d  an  [cirf2  4-  CaTj2  4-  +  C4T4 J]1^3  : 

d  =  [.61295823C— .19352088)2  +  ,00682012(.11636829)a  4- 

.34271100(  .29113385)2  4-  .037510Q6( .48124468)2]  ^a 

.24654207 

The  vector  of  scores  (Nair,  1986,  1987)  is  then  obtained  by  lk  =s  rk/d  : 
lk  -0  7849  0.4720  1.1809  1,9520 

Shifting  the  scores  so  that  the  score  for  a  normal  implantation  (fetus)  is  zero,  the  final 
scores  are: 

l;  0.0000  1.2569  1.9658  2.7369 

Then,  a  litter  with  11  implantations  of  which  4  are  classified  as  normal,  1  as  growth  re¬ 
tarded,  5  as  malformed  and  1  dead,  would  have  e  severity  index  of: 


SI=[0. 0000(4)4-1. 2569(1)+1.9658(5)+2.7309(1)]/11=1, 2566 

This  can  be  interpreted  in  light  of  the  above  scores,  i.e.,  an  index  near  zero  would  be 
indicative  of  a  litter  with  nearly  all  normal  fetuses  at  cesarean-section  and  a  score  near 
2,7369  would  be  indicative  of  a  litter  with  nearly  all  fetuses  dead  at  cesarean-section. 

Designs  fot  the  Analysis  of  the  Severity  Index 

Five  designs  were  evaluated  which  assume  normality.  The  one-way  classification,  a 
one-way  classification  using  litter  size  as  a  covariate,  a  generalized  randomized  block  using 
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litter  size  as  a  blocking  variable,  and  a  weighted  analysis  using  in  one  case  litter  size  as  a 
weight  and  in  another  the  square  root  of  litter  size.  The  results  are  summarized  in  Table 
IV  in  terms  of  calculated  F,  associated  P  values  and  JJ2. 


Table  XV 


One  Way  analysis 
Covariance 
Generalized  RBD 
Weighted  AOV  (litter  size) 

Weight  AOV  (^littersize) 


f 

F 

P 

R? 

3,81 

62.97 

< 

.0061 

.46 

3,80 

25.99 

< 

.0001 

.53 

3,65 

20.52 

< 

.0001 

.58 

3,81 

15.62 

< 

.0001 

.37 

3,81 

33.11 

< 

.0001 

.55 

As  was  expected  the  covariance  and  blocking  provided  an  improvement  over  the  one* 
way  classification  as  measured  by  J22.  However,  the  magnitude  of  the  improvement  does 
not  seem  to  warrent  the  chance  of  violating  the  more  restrictive  assumptions  placed  on  the 
experiment  by  those  designs.  A  better  alternative,  in  the  parametric  case,  may  be  using 
the  square  root  of  litter  size  as  a  weight  which  provides  nearly  the  same  value  of  R 2  as 
does  the  blocking  design.  However,  we  would  prefer  the  one-way  analysis  for  its  simplicity 
and  robustness  in  application. 

The  normality  assumption  on  the  severity  index  is  quite  suspect  in  many  situations. 
As  an  alternative,  the  nowparametric  Kruskal- Wallis  procedure  was  carried  out.  In  view  of 
the  overwhelming  significance  of  the  parametric  procedures,  this  result  was  not  surprising 
X*  a  47.75;  af  *  3  p  <  .0001.  Figure  1  compares  the  linearity  of  the  mean  severity  index 
and  the  median  severity  index. 


Figure  1 


STAT 


DOSE  (mg/kg/doy) 
o  MEAN  0  MEDIAN 
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Statistical  Procedure 


The  consideration  of  litter  size  is  not  necessary  for  analysis  of  the  Si’s,  It  is  important 
to  note  than  the  Si’s  are  probably  not  normally  distributed,  particularly  in  the  control 
group  and  at  the  higher  dose  levels.  The  following  is  suggested  for  toxicity-teratogenicity 
studies. 


1.  If  the  Si's  are  reasonably  normal,  calculate  the  AOV  F-statistic  for  a  one  way  layout, 
Use  this  statistic  to  test  for  differences  in  location. 

2.  If  F  is  significant,  follow  with  linear  constrasts  to  test  for  increasing  trend. 

3.  If  significant  use  Dunnett’s  procedure  to  compare  control  mean  with  each  of  the 
treatment  means  to  establish  no-effect  leve, 

4.  In  the  presence  of  non  normality  use  a  similar  sequence  of  nonparametric  test,  e.g. 
K  -  W,  Jonckheere/Terpstra,  and  Dunn’s  procedure. 


SAS  code  is  available  which  reads  litter  data,  calculates  scores,  computes  Si’s  and  cal¬ 
culated  the  statistics.  The  results  above  have  been  "tested”  by  simulation  analysis  of 
additional  nitrofen  studies  and  two  other  biological  examples.  Also,  the  method  detects 
different  dose  patterns  with  equal  ability,  The  K  -  W  test  showed  consistently  higher  power 
than  the  F-statistic  in  the  simulation  studies, 
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Abstract 

The  Anderson-Darling  goodness-of-flt  procedure  emphasizes  agreement 
between  the  data  and  the  hypothesized  distribution  in  the  extremes  or  tails. 
An  Improved  table  of  the  quantiles  of  the  Anderson-Darling  statistic,  useful 
for  small  sample  sizes,  was  constructed  using  the  Cray-2  supercomputer.  The 
power  of  the  Anderson-Darling  test  is  compared  to  the  Kolmogorov  and  the 
Cramer- von  Mlses  tests  when  the  null  hypothesis  is  the  normal  distribution 
and  the  alternative  distributions  are  the  Cauchy,  the  double  exponential,  and 
the  extreme  value  distributions. 
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1.  INTRODUCTION 


Consider  a  random  sample  Xlf  X2,  ...» Xn  from  a  population  with  a 
continuous  distribution  function.  One  method  of  testing  the  hypothesis  that 
the  n  observations  come  from  a  population  with  a  specified  distribution 
function  F(x)  is  by  a  chi-square  test.  This  test  requires  a  subjective 
partitioning  of  the  real  line  R  and  a  comparision  of  the  empirical  histogram 
with  the  hypothetical  histogram.  A  more  objective  method,  is  to  compare  the 
empirical  distribution  function  Fn(x)  with  the  hypothetical  distribution 
function  F(x).  The  empirical  distribution  function  based  on  n  observations  is 

defined  as  Fn(x)  «•  —  if  exactly  k  observations  are  less  than  or  equal  to 

x,  for  k  ■■  0,  1,  ...,  n. 

To  compare  the  empirical  and  hypothetical  distribution  functions  a 
measure  of  their  difference  is  required.  Addressing  this,  Anderson  and  Darling 
[1952]  considered  the  following  metrics  in  function  space: 

W.'-„7  [Fn(x)  —  F(x)|2  tflFMI  dF(x)  (1.1) 

— OO 

and 
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I<n  —  sup  V7lFn(x)-F(x)|Vi?iFOO)  .  (1.2) 

-00  <  X  <  00 

Samples  producing  large  values  of  W„  (or  Kn)  lead  to  rejection  of  the  null 
hypothesis  that  the  population  distribution  function  is  F(x).  One  of  the 
contributions  of  Anderson  and  Darling  was  the  Incorporation  of  a  non-negative 
weight  function  ip  in  (1.1)  and  (1.2).  By  a  suitable  choice  for  ip,  specific  ranges 
of  values  of  the  random  variable  X,  corresponding  to  different  regions  of  the 
distribution  F(x),  may  be  emphasized.  For  i/.>[F(x)]  m  1,  W;j  becomes  the 
Cramer- von  Mises  statistic  (Cramer,  1928  and  von  Mises,  1931]  and  Kn 
becomes  the  Kolmogorov  statistic  [Kolmogorov,  1933]. 

The  tails  of  the  distribution  function  will  be  accentuated  in  the 
investigation  detailed  in  this  paper;  Anderson  and  Darling  suggest  using 

x!|_ _ ! _ 

F(x)|l-F(x))  ' 

With  this  choice  for  the  weighting  function,  metric  (1.1)  becomes  the  basis  for 
the  Anderson-Darling  statistic. 

In  Section  2,  the  Anderson-Darling  test  statistic  is  developed:  in 

Section  3,  the  most  accurate  tabulation  to  date  of  the  test  statistic  is  provided. 

In  Section  4,  the  description  and  the  results  of  a  power  study  are  given  in 
which  the  Anderson-Darling,  the  CramSr-von  Mises,  and  the  Kolmogorov 

statistics  are  compared. 


119 


2.  THE  ANDERSON-DARLING  STATISTIC 


For  a  fixed  value  of  the  random  variable  X,  say  X  ■»  x,  the  empirical 
distribution  function  Fn(x)  is  a  statistic,  since  it  is  a  function  of  the  sample 
values  xlf  x2,  xn.  The  distribution  of  this  statistic  is  established  as  a  lemma. 


Lemma  (2.1):  If  Fn(x)  is  the  empirical  distribution 
function  corresponding  to  a  random  sample  Xlt  X2,  Xn 
of  size  n  from  a  distribution  H(*),  then  for  a  fixed  x,  nFn(x) 
is  distributed  binomial  (H(x),n). 

Proof: 

P(nF„(x)  —  k)  ■■  P(  exactly  k  values  Xj  <  x),  for  k  0,  1,  n. 

Let  Z|  —  I(_00|  xj  (Xj),  where  the  indicator  function  I  is  defined  as 

1,  if  —oo  <  Xj  <x 

I(-oo,  x]  (Xj)  -  ' 

0,  otherwise  . 

s 

Then  EZj  counts  the  number  of  sample  values  xt  <  x. 

Here  each  Zj -'-Bernoulli  (H(x)),  so  EZi^binomial^x)^). 
Therefore, 

P(nF'n(x)  ■  k)  m  P(exactly  k  values  x,  <  x) 
-P(EZi-k) 
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From  Lemma  2.1, 


E(F„(x)|-^E[nFn(:<)]-H(x) 

and 

Var[Fn(x)]  -  ±  Var  [tiF„(x)]  -  ±  H(x)  [1  -  H(x)]  .  (2.1) 

To  assist  in  the  determination  of  a  suitable  weighting  function  i{-), 
that  is,  a  function  that  will  weight  more  heavily  values  in  the  tails  of  the 
distribution  F(x)  at  the  expense  of  values  closer  to  the  median,  consider  the 
expectation  of  the  squared  discrepancy  E  |^Fn(x)  —  F(x)j8.  It  is  important  to 

keep  in  mind  that  the  value  x  is  fixed,  so  F(x)  is  a  constant,  and  the 
expectation  is  with  respect  to  the 'random  variable  Fn(x)  whose  distribution  was 
established  in  Lemma  2.1.  Then 

n  E  [Fn(x)  -  F(x)  2  -  n  E  [Fn(x)  -  H(x)  +  H(x)  -  F(x)  2 

0 

■ 

(x)  -  H(x)  • 

which,  after  algebraic  manipulation  (Appendix  A)  yields  the  variance  and  bias" 
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Under  the  null  hypothesis  H0:  H(x)  •»  F(x)  Vx,  (2.2)  becomes 

n  E  [f„(x)  -  F(x)]4  -  F(x)[l  -  F(x)| .  (2.3) 

Anderson-Darling  chose  as  a  weighting  function,  i/>[F  (x)]  «  j  ^  ~  F(x))  ' 

Weighting  by  the  reciprocal  of  (2.3)  takes  into  consideration  the  variance  of 
the  statistic  Fn(x)  and  also  maintains  the  objective  of  accentuating  values  in 
the  tails  of  F(x). 


With  this  choice  of  weighting  function  and  without  loss  of  generality 
assuming  Xi  <  x2  <  ...  <  xn,  let  F(x)  ■■  u,  dF(x)  du,  and  F(Xj)  ■■  U|.  Then 
the  Anderson-Darling  test  statistic  (2.4)  can  be  rewritten  as  expression  (2.5)  by 
expansion  and  integration  (Appendix  B). 


7  1F,(X)-F(x)]a 
“-•Lf(x)  [1  —  F(x)] 


dF(x) . 


(2  A) 


Wj  -  -  n  -  ~  2  f(2j— l)ln  u,  +  (2(n-J)+l)  ln(l— u,)l .  (2.5) 

11  j-1  L  J 
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3.  DISTRIBUTION  OF  THE  ANDERSON-DARLING  STATISTIC 


The  asymptotic  distribution  of  was  derived  by  Anderson  and 
Darling  [1952].  Lewis  [1961]  undertook  the  tabulation  of  F(z;  n)  —  P(W“  <  z) 
for  n  —  1, 2,  8  and  for  incremental  values  of  z  over  the  interval 
[0.025, 8.000].  Lewis’  table  entries  were  computed  using  a  Monte  Carlo 
procedure  to  generate  an  empirical  approximation  Fm(z;n)  to  the  distribution 
function  F(z;n)  based  on  rn  samples  of  size  n.  At  that  time,  computational 
restrictions  essentially  limited  the  accuracy  of  the  table  entries  to  within 
0.00326  of  the  true  value. 

Following  an  analogous  procedure  based  on  expression  (2.5)  and  the 
observation  that  the  Uj  are  distributed  U[0,l]  [Feller,  1966],  the  table  appearing 
in  Lewis’  paper  was  recalculated  using  a  Cray-2  supercomputer.  A 
Kolmogorov-type  bound  [Conover,  1980]  was  used  to  construct  a  9 5^ 
confidence  band  for  .he  distribution  function  F(z;n). 

In  general,  the  width  of  a  (1  —  a)100%  confidence  band  is  equal  to 
twice  the  value  of  the  (1  —  a)100%  quantile  of  the  Kolmogorov  statistic 

I<m-  sup  Vin  lFm(x)  —  F(x)l  ,  where  m  is  the  number  of  sample 

-oo  <  x  <  oo 
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values  used  in  the  construction  of  Fm(x).  With  n  fixed*  the  05%  confidence 
band  can  be  made  arbitrarily  small  by  a  suitable  choice  for  m,  the  number  of 
Monte  Carlo  samples.  The  commonly  tabled  [Miller,  1950]  asymptotic 
approximation  for  the  05th  quantile  is  1.358/Vm  .  However,  Harter  [1080] 
suggests  using 


1.358 


m  + 


3.5 


where  r  —  (m  4-  4) 


(3.1) 


for  an  Improved  approximation. 


Using  approximation  (3.1)  to  construct  a  95%  confidence  band  with 
the  width  not  exceeding  0.001,  the  value  for  m  must  be  at  least  7,375,881.  In 
this  simulation,  m  was  chosen  to  be  7.4  million.  Table  1  lists  the 
reconstruction  of  Lewis'  table,  now  accurate  within  0.0005.  Again,  z  ranges 

from  0.025  to  8.000  and  for  n  -  1,  2 . 10.  The  column  labeled  "oo"  contains 

the  asymptotic  values,  rounded  to  four  decimal  places. 


124 


TABLE  1.  Quantiles  cf  F(z;n)  for  n  =  1  thru  10  and  oo  . 
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0.6250  0.1607  0.3936  0.3922  0.3870  03841  0.3827  0.3812  0.3809  0.3803  0.3798  0.3753 

0.6500  0.4811  0.4166  0.1143  0.4096  0.4069  0.4052  0.4038  0.4035  0.4029  0.4024  0.3980 

0.6750  0.5007  0.4391  0.4357  0.4314  0.4286  0.4270  0.4256  0.4254  0.4248  0.4244  0.4199 

0.7000  0.5188  0.4618  0.4563  0.4524  0.4495  0.4481  0.4467  0.4464  0.4459  0.4456  0.4412 


TABLE  1.  continued 
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2.0000  0.8018  0-9015  0.9038  0.9051  0.9058  0.9061  0.9064  0.9066  0.9068  0.9071  0.9082 

2.0500  0.9002  0.9070  0.9093  0.9107  0.9113  0.9117  0.9120  0.9121  0.9124  0.9127  0.9138 

2.1000  0.9053  0.9122  0.9146  0.9159  0.9165  0.9169  0.9172  0.9174  0.9177  0.9179  0.9190 

2.1500  0.9102  0.9171  0.9195  0.9208  0.9214  0.9218  0.9221  0.9223  0.9226  0.9228  0.9239 

2.2000  0.9148  0.9218  0.9241  0  9254  0.9259  0.9264  0.9266  0.9268  0.9272  0.9273  0.9285 


TABLE  1.  continued 
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0.9807 

0.981b 


TABLE  1.  continued 
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4.  POWER  STUDY 


The  power  of  the  Anderson-Darling  test  was  compared  with  two  other 
goodness-of-fit  procedures  based  on  the  empirical  distribution  function:  the 
Kolmogorov  and  the  Cram^r-von  Mises  tests.  The  Kolmogorov  statistic 
introduced  in  Section  1  as  metric  (1.2)  with  weighting  function  4>  [F(x)]  ■  1 
becomes 


Kn  -  sup  Vn  lFn(x) -F(x)l  .  (4.1) 

-00  <  X  <  00 


For  an  ordered  sample  xt  <  x2  <  •••  <  xn  and  F(xj)  -  Uj,  (4.1)  may  be 
evaluated  using  D  —  max(  D+,  D"  ),  where 


D+  - 


max 

» 

l 


- Ui 


and 


D“  »  max 


Ui  ~ 


i  -  1 


The  Cramer- von  Mises  statistic,  defined  as 


W2 

yvn 


oc 


/  (F„(x)  -  F(x)|2  dF(x) 


can  be  reduced  to  (4.2)  for  ease  of  computation  (Appendix  C); 
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n 

£ 

i-i 


Uj  - 


2i  -  1 
2n 


1 

12n  ' 


(4.2) 


In  the  power  study,  two  cases  were  considered.  Case  1  corresponds  to 
the  situation  in  which  the  parameters  of  the  hypothesized  distribution  are 
completely  specified.  Case  2  corresponds  to  the  situation  in  which  the 
parameters  are  not  specified  and  must  be  estimated  from  the  sample  data. 

For  both  case  1  and  2,  the  null  hypothesis  is 

H0:  A  random  sample  Xt,  X2,  Xn  comes  from  a  normal  population 
or 

H0:  H(x)  —  F(x),  where  F(x) N(/u,  tf2)  . 

As  alternative  hypotheses,  the  Cauchy,  double  exponential,  and  extreme  value 
distributions  were  chosen,  each  with  location  parameter  the  same  as  the  null 
hypothesis.  This  provided  a  heavy-tailed,  light-tailed,  and  skewed  distribution, 
respectively,  against  which  the  power  of  the  three  goodness-of-ftt  tests  are 
compared. 

The  power  functions  do  not  exist  in  closed  form;  they  are 
approximated  empirically  via  a  Monte  Carlo  simulation.  To  determine  a  point 
on  the  power  curve,  a  large  number  of  samples  of  size  n  was  generated  from  a 
specific  distribution  serving  as  the  alternative  hypothesis.  The  number  of 
times  the  null  hypothesis  was  rejected  at  a  specific  level  of  significance  was 
recorded.  The  ratio  of  the  number  of  rejections,  Y,  to  the  total  number  of 
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samples  generated,  N,  provides  an  estimate,  p  “Y/N,  of  the  probability  of 
rejecting  the  null  hypothesis  when  it  should  be  rejected  (power).  The  value  p 
determines  a  point  on  the  power  curve  corresponding  to  a  specific  sample  size 
n,  a  specific  significance  level  a,  and  a  specific  alternative  hypothesis. 


To  determine  the  number  of  samples  of  size  n  required  for  a 
sufficiently  accurate  estimate  of  p,  a  nonparametric  technique  was  employed. 
Since  the  counter  Y  is  distributed  binomial(  •  ;  p.N)  where  the  parameter  p  is 
the  true  but  unknown  power,  and  since  an  approximate  confidence  Interval  for 
p  can  be  constructed  [Conover,  1980]  using 


1  —  a  cat  P 


O'  '  Q  ' 


)/N 


(4.3) 


samples  of  size  n  continued  to  be  generated  from  the  alternative  distribution 
until  the  confidence  interval  for  p  given  in  (4.3)  was  sufficiently  small. 


The  confidence  interval  coefficient  1  —  a  was  chosen  to  be  0.975  and 
the  confidence  interval  width  not  to  exceed  0.025.  Then  the  confidence  limits 
(4.3)  were  successively  evaluated  until  the  interval  width  was  satisfied. 
Considering  a  "worst-case"  scenario  in  which  p  —  1/2  and  the  variance  of  its 
estimate  p  is  greatest,  equating  Y/N  —  1/2  in  (4.3)  suggests  that  samples  of 
magnitude  8037  might  be  required.  A  minimum  'alue  for  N  of  100  was 
imposed  to  prevent  premature  termination  of  the  procedure. 
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4.1.  Case  1:  Distribution  Parameters  Specified. 

The  power  study  for  case  1  specified  the  parameters  of  the 
hypothesized  distribution  as  N(0,1).  The  results  of  the  study  are  summarized 
in  Figures  1  —  12.  For  each  of  the  three  distributions  serving  as  an  alternative 
hypothesis,  samples  of  size  n  —  5,  10,  15,  20  were  chosen  for  study  and,  as 
previously  mentioned,  the  location  parameters  of  both  the  null  and  alternative 
hypotheses  coincided.  The  scale  parameter  for  the  alternative  hypothesis  were 
values  from  0.025  to  3.000  in  increments  of  0.025. 

The  level  of  significance  for  the  study  was  0.05.  The  critical  value  for 
each  test  was  determined  from  tables  In  Conover  [1980]  for  the  Kolmogorov 
test,  Stephens  and  Maag  [1908]  for  the  Cramer-von  Mises  test,  and  Table  1  in 
Section  3  of  this  paper  for  the  Anderson-Darling  test. 

The  Anderson-Darling  test  demonstrated  overall  superiority  for  the 
sample  sizes  and  hypotheses  chosen  for  this  study.  This  is  perhaps  to  be 
anticipated  in  view  of  tne  emphasis  on  agreement  in  the  tails  by  the 
Anderson-Darling  procedure,  but  the  magnitude  of  difference  over  the 
Kolmogorov  and  Cram^r-von  Mises  tests  is  -repressive. 

The  power  curves  corresponding  to  n  —  10,  15,  20  ar«  distinguished 
by  their  characteristic  of  decreasing  to  a  global  minimum  before  becoming 
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monotone  increasing.  An  explanation  of  this  feature  is  suggested  by 
consideration  of  Figures  13—15  in  which  the  distribution  functions  of  the 
N(0,1)  and  Cauchy  (0,f)  are  compared.  There  it  is  seen  (Figure  14)  that 
corresponding  to  f  ■  0.50  the  two  distribution  functions  are  similar;  an 
increase  (decrease)  in  the  scale  parameter  f  causes  the  tails  of  the  distributions 
to  become  more  distinct.  Values  in  a  neighborhood  of  f  as  0.50  marked  the 
global  minimum  throughout  the  study. 

4.2.  Case  2:  Distribution  Parameters  Estimated. 

The  Andorson-Darling,  Kolmogorov,  and  Cram^r-von  Mises 
goodness-of-fit  tests  were  developed  for  use  in  case  1  where  distribution 
parameters  are  specified,  and  so  precludes  their  use  in  the  more  likely  situation 
where  parameters  must  be  estimated.  In  practice,  these  procedures  are 
sometimes  used  anyway  with  the  caveat  that  the  tests  are  likely  to  be 
conservative.  Stephens  [1974]  provides  adjustments  to  the  test  statistics  that 
enables  the  tests  to  be  used  to  test  the  assumption 
H0:  H(x)  ■■  F(x),  where  F(x)  N(/i,  a2)  and  the  population  parameters  are 
estimated  from  the  data. 

The  results  of  the  power  study  for  case  2,  are  summarized  in 
Figures  16  —27.  As  in  case  1,  the  sample  sizes  are  n  ■*  5,  10,  15,  and  20.  and 
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the  level  of  significance  is  0.05.  Both  location  and  scale  parameters  coincide; 
the  scale  parameter  are  values  from  0.025  to  3.000  in  increments  of  0.025. 

The  power  plots  are  horizontal,  demonstrating  that  power  does  not 
change  with  scale  parameter  and  provides  empirical  support  for  Stephens’ 
transformations.  Power  Increases  with  increasing  sample  size,  as  would  be 
expected.  When  both  location  and  scale  parameters  agree,  all  three  tests  are 
competitive  for  the  sample  sizes  and  alternative  distributions  chosen  for  this 
study. 
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Cauchy  vs  N(0,1) 


FIGURE  1.  Power  Curve  for  Cauchy  Generated  Data,  n 


Double  Exponential  vs  N(0,1) 
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FIGURE  5.  Power  Curve  for  Double 
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Double  Exponential  vs  N(0,1) 
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FIGURE  7.  Power  Curve  for  Double  Exponential  Generated  Data,  n  = 


Double  Exponential  vs  N(0,1) 


FIGURE  8.  Power  Curve  for  Double  Exponential  Generated  Data,  n 


Extreme  Value  vs  N(0,1) 
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FIGURE  9.  Power  Curve  for  Extreme  Value  Gei 


»eme 


'rated  Data, 


Extreme  Value  vs  N(0,1) 


FIGURE  11.  Power  Curve  for  Extreme  Value  Generated  Data,  n 


Extreme  Value  vs  N(0,1) 


FIGURE  12.  Power  Curve  for  Extreme  Value  Generated  Data, 


Normal(0,l)  &  Caucby(0,0.18) 


FIGURE  13.  Cumulative  Distribution  Curves,  Normai(0,l)  &  Cauchy(0,0.18) 


[RE  14. 


Normal(0,l)  &  Cancliy(0,l) 


FIGURE  15.  Cumulative  Distribution  Curves,  Normal(Q,l)  &  Cauchy (0,1) 


Cauchy  vs  N(x, 


FIGURE  16.  Power  Curve  for  Cauchy  Generated  Data, 


Cauciiy  vs  N(x,  s2) 
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FIGURE  17.  Power  Curve  for  Cauchy  Generated  Data,  n  =  10 


Cauchy  vs  N(x,  sa) 


FIGURE  19.  Power  Curve  for  Cauchy  Gei 


Double  Exponential  vs  N(x,  s2) 
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FIGURE  20.  Power  Curve  for  Double  Exponential  Generated  Data,  n  = 
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0.4 


Double  Exponential  vs  N(x,  s2) 
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Extreme  Value  vs  N(x,  s2) 


FIGURE  24.  Power  Curve  for  Extreme  Value  Generated  Data,  n 


Extreme  Value  vs  N(x,  s2) 


FIGURE  25.  Power  Curve  Cor  Extreme  Value  Generated  Data,  n  =  10 


Extreme  Value  vs  N(x,  s2) 
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CURE  26.  Power  Oirvs  for  Extreme  Value  Generated  Data,  n  =  15 


Extreme  Value  vs  N(x,  s2) 


FIGURE  27.  Power  Curve  for  Extreme  Value  Generated  Data, 
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APPENDIX  Ai  EXPECTION  OF  SQUARED  DISCREPANCY 
BETWEEN  AN  EMPIRICAL  DISTRIBUTION  FUNCTION 
AND  A  SPECIFIED  DISTRIBUTION  FUNCTION 


To  help  find  a  suitable  weighting  function,  ^(x)],  lets  look  at 
nE[Fn(x)-F(x)]8. 


n  E  [f„(x)  -  F(x)]2  -  n  E  [f„(x)  -  H(x)  +  H(x)  -  F(x)]‘ 

(f„(x)  -  H(x)  -  '  F(x)  -  H(x) ' 
\  /  '  -I 


n  E 


-  n  E 


F„(x)  -  H(x) 


F„(x)-H(x) 


F(x)  —  H(x)  r 


+ 


F(x)  -  H(x) 


r  ' 

-  n  E  [f„(x)  -  H(x)]S  -  2  '  F(x)  -  H(x)  E  [f„(x)  -  H(x)] 


+  F(x)  -  H(x) 


r 


n  F  [f n(x)  —  H(x)]2  +  F(x)  —  H(x) 
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n  E 


n  E  [fb(x)  2  -  2  Fn(x)H(x)  +  H2(x)]  +  |f(x)  -  H(x) 


-  n  E  |fb(x)  3]  -  2  H(x)  E  [fb(x)]  +  H2(x)  + 


F(x)  -  H(x) 


—  H(x){l  -  H(x)}  +  H2(x)  -  2  H2(x)  +  H2(x) 


|f(x)  -  H(x) 


[f„(x)-F(x)]  -■> 


H(x){l  -  H(x)} 


F(x)  -  H(x) 


Under  the  null  hypothesis,  l.e.,  H©:  H(x)  —  F(x), 


i>e[f,,(x)-F(x)]J-F(x)[1-F(x)]. 
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APPENDIX  Bi  EXPANSION  AND  INTEGRATION  OF 
THE  ANDERSON-DARLING  STATISTIC 


w..„  j  lF-W-FWI‘  mx) 

*  i.  f(*)[i-fw] 

7  f.w-fwi8  »7'iFn(x)-Fwia 

.4  F(x)[l  —  F(x)]  dFW  +  kS  /  F(x)[l  —  F(x)]  dF(x) 


—  n 


°®  [Pn(x)  -  F(x)l2  ,  4 
+  {  F(x)|l  -F(x)]  dF<X) 


Let  F(x)  «■  u 
dF(x)  -  du 
F(xi)  -  ui 


n-1  Ufc+‘ 


k_  _ 
n 


I  f  JQ^i2 .  du + y  7'  du  +  f  . 

[o  “  U  ~  “)  k?i  1  “  (I  -  «)  u0  u  (I  —  u) 


n 


n-l 


Uk+l 


k2  2k  ,  -j 

— r - u  + 

n2  n 


-u,  u.)  +  s  /  -  u) 


du 


-1  +  un  -  In  un] 


du 
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ux  —  In  (1 


Uj  —  In  (1 
n-k 

+  E 

k-l 

n-1 

+  S 

k-l 

n-1 

+  E 

k-l 

+  [- 


n-1 

-  ».)  +  E 

k-l 


k3  y  du 
a3  £  »  (1  -  u) 


n-1 

+  E 

k-l 


2k 

n 


u 


Uk+l 

£  U  (1  -  U) 


du 


n-1 

+  E 

k-l 


uIm-i  2 

U* 


/ 


Uk  u  (1  -  u) 
+  [“1  +  un  -  In  un] 


du 


-Ui) 

■jj  [in  (Uk+l)  -  In  (1  -  Ujc+i)  -  In  uk  +  In  (1  - 
[ln  (!  -uk+1)  -  In  (1  -uk)] 

[  “  uk+i  -  In  (1  -  uk+1)  +  uk  +  In  (1  -  uk)] 

1  +  un  -  ln  un]| 
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—  In  (1  —  Ui)  +  £l  -  (2-~-  flnuk-!n(l -uk)l 
k-i  {  n  L  i 

+  ~|  [in  un  —  In  (1  —  un) ] 


*  ■ 

+  I?  -  ~  In  (1  ~  «k) 

k-l  L  n 

+  l»  (1  "  «.) 


+  Ui  +  In  (1  -  Uj)  -  un  -  In  (1  -  Un) 

+  [  — 1  +  un  —  In  unj 

1  -  S  [N^]  'n  nk  +  (ate=a±l| ,»  (1  -  „k)l } 
k-i  LI  n  J  n 

“  S  [(2k  -  1)  In  uk  +  (2  (n — k)  +  1)  In  (1  -  uk)]  . 
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APPENDIX  Ci  DERIVATION  OF  THE 
CRAMfiR-VON  MISES  STATISTIC 


For  an  ordered  sample  Xj  <  x2  <  •  •  ■  <  xn  the  empirical  distribution 
function  is  defined  as 


Fn(x)  - 


0  :«  <  xx 

—  for  xk  <  x  <  xk+1  . 
1  xn  <  x 


The  Cram^r-von  Mises  statistic  may  be  written 


00 


■>  J  |Fn(x)  -  F(x)]adF(X) 


-00 


w  n 


Xt  n—1 

/  |F„(x)  -  F(x)]a  dF(x)  +  £  /  [F„(x)  -  F(x)|a  dF(x) 

-oo  k-1  Xk 


00 


+  /  [Fn(x)  -  F(x)]2  dF(x) 


Let  F(x)  ■■  u 
dF(x)  -  du 
F(x,)  -  Uj 
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Then 

n  /  |Fn(x)  -  F(x)]3dF(x) 
-00 


■  «  It  »i  +  -7  s'  ka(uk+l  -  Uk)  -  7  °S  k  (“2-*.  - »{) 

[3  n3  k.,  n  k.i 

•  ■ 

+  7  “e  (ug+1  ~  uf)  +  J  -  un  +  u2  - 
3  k-i  l*  0 

-  a  It  U1  +  "V  El  (  ~  (2k  “  !)uk)  +  (a  ~  l)4  un 
3  a  Lk-1 

-  -  f-El(uk2)  +  (n-l)uft2 
11  k-1 


1 70 


Completing  the  square, 
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Nonpare,  a  Consultation  Systeir  for  Analysis  of  Data 


J.C.  Dinner  III 
T.P.  Hanratty 
M.S.  Taylor 

US  Army  Ballistic  Research  Laboi  atory 
ATTNi  SLCBR-SE-P 

Aberdeen  Proving  Ground,  MD  21005-5066 


Abstract.  Nonpare,  a  consultation  system  for  analysis  of  data  using  nonparametric 
statistical  procedures,  is  under  active  development.  It  is  intended  to  serve  as  an 
intelligent  interface  that  will  act  as  a  guide,  an  instructor,  and  an  interpreter  to  a  body  of 
statistical  software.  Nonpare  exists  as  a  prototype,  with  a  limited  release  planned  in  1989 
for  field  testing. 
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1.  Introduction 

Statistical  software  packages,  to  large  extent,  accept  any  properly  configured 
data  set  and  proceed  to  process  it.  Few  if  any  checks  are  made  to  ensure  the  adequacy 
of  the  data  and  the  suitability  of  the  analysis,  and  little  is  done  to  provide  an 
explanation  or  interpretation  of  the  results.  This  requires  a  great  deal  from  the  user. 
Declining  computation  costs,  together  with  increased  availability  of  computers  and 
proliferation  of  statistical  software,  has  further  enhanced  the  opportunity  for  faulty 
data  analysis.  Application  of  expert  system  techniques  from  artificial  intelligence  to 
produce  more  cognizant  software  is  one  approach  to  reversing  this  unfortunate  trend. 

In  1985,  a  workshop  sponsored  by  AT&T  Bell  Laboratories  brought  together 
many  of  the  active  investigators  in  artificial  intelligence  and  statistics  and  was  the 
genesis  of  a  book  by  the  same  title  edited  by  Gale  [1],  This  reference  is  in  essence  the 
proceedings  of  the  workshop;  but  the  papers  given  there,  some  with  extensive 
bibliographies,  provide  the  most  complete  centrally-located  account  of  research  in  this 
topic  to  date. 

This  report  details  an  effort  underway  at  the  US  Army  Ballistic  Research 
Laboratory  (BRL)  to  develop  a  consultation  system  for  analysis  of  data  using 
nonparametric  statistical  procedures.  The  system,  called  Nonpare,  is  intended  to 
serve  as  an  intelligent  interface  that  will  act  as  a  guide,  an  instructor,  and  an 
interpreter  to  a  body  of  statistical  software.  The  system  is  currently  a  prototype,  with 
a  first  release  planned  for  1989  for  field  testing. 

2.  Nonpare 

Nonparametric  statistics  is  too  large  an  area  to  hope  to  encompass  at  once, 
especially  if  the  entire  field  of  mathematical  statistics  is  partitioned  into  parametric 
and  nonparametric  procedures.  The  common-sense  approach  to  construction  of 
consultation  systems  suggests  limiting  the  domain  of  application,  but  nonparametric 
statistics  has  qualities  that  make  it  strongly  appealing. 

Nonparametric  data  analysis  is  characterized  chiefly  by  the  absence  of 
restrictive  distribution  assumptions  -  notably  freedom  from  dependence  on  the  normal 
(Gaussian)  distribution.  Many  nonparametric  statistical  procedures  are  exact  rather 
than  approximate  for  small  data  sets,  and  they  are  the  only  confirmatory  procedures 
which  can  be  used  to  analyze  data  collected  on  a  nominal  or  an  ordinal  scale  of 
measurement.  For  these  and  other  compelling  reasons  advanced,  for  example,  by 
Conover,  [2]  Hollander  and  Wolfe,  [3]  and  Lehmann,  [4]  nonparametric  procedures 
find  use  in  a  wide  variety  of  disciplines. 

2.1  The  System  Structure 

Nonpare  uses  Genie,  an  expert  system  shell  developed  at  the  BRL,  [5]  to 
provide  a  frame-based  production  system  with  forward  and  backward  inferencing  as 
well  as  an  explanation  facility  that  allows  the  user  to  interrogate  the  system -what 
hypotheses  arc  being  entertained,  what  rules  are  being  verified,  what  facts  are  in 
evidence.  Genie  was  chosen  over  commercial  expert  system  shells  for  the  research 
and  development  of  Nonpare  because  of  its  accessibility  for  modification. 
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Nonpare,  shown  schematically  in  figure  1,  consists  of  three  subsystems  in 
addition  to  Genie. 


Genie 


Figure  1.  Nonpare  system  overview. 

The  system  dictionary  is  a  facility  whose  purpose  is  to  provide  on-line 
explanation  of  statistical  jargon  that  may  appear  during  the  interactive  dialog  between 
Nonpare  and  the  user.  Expert  domain  knowledge,  codified  in  English-like  rules, 
resides  in  the  knowledge  base.  Once  an  appropriate  procedure(s)  has  been  identified, 
the  data  are  analyzed  and  the  results  explained  by  the  nonparametric  data  analysis 
component.  Graphics  is  used  to  summarize  the  data  and  enhance  the  explanation.  In 
total,  the  user  is  led  within  system  limitations  to  an  appropriate  statistical  procedure 
through  an  interactive  process  in  which  the  user  is  questioned  and  can  in  turn  question 
the  consultation  system.  Nonpare  is  written  in  Interlisp-D  and  currently  runs  on 
Xerox  1100  Series  Lisp  machines. 

3.  An  Illustrative  Session 

Following  the  dictum  of  American  educator  John  Dewey  (I85‘M‘)52)  that 
"We  learn  by  doing,"  a  detailed  session  with  Nonpare  follows,  in  which  the  main 
system  features  are  illustrated. 

Example  3.1 

Suppose  that  a  ballistician  needs  to  assess  the  effectiveness  of  a  newly 
designed  kinetic  energy  penetrator  against  a  specific  armor  plate.  In  particular,  the 
experimenter  wouid  like  to  establish  whether  the  probability  of  perforation  exceeds 
.80  a  level  already  attained  with  existing  technology.  Fourteen  rounds  are  fired,  and 
[perforation  and  (n)onperforation  recorded  to  obtain;  n,  p,  p,  p,  n,  p,  p,  p,  p,  n,  p,  p, 
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p,  p.  Is  the  Pr{  perforation}  >  .80  ? 

(A  diversion  here.  Searching  for  a  statistical  procedure  with  a  set  of  data 
already  collected  is  precisely  how  not  to  proceed.  The  purpose  for  collecting  the  data 
should  first  be  established,  and  then  the  statistical  tools  available  to  support  this 
purpose  determined.  Then  the  collection  and  analysis  of  data  can  proceed  in  an 
informed  manner.  Lamentably,  the  methodology*  search  scenario  is  enacted  over  and 
over  again;  so  this  example  is  not  too  contrived.) 

It  should  be  apparent  from  the  onset  that  the  question  regarding 
Pr{  perforation}  >  .80  can  never  be  answered  unequivocally  yes  or  no,  but  only  with 
some  degree  of  qualification. 

Nonpare  presently  has  nineteen  distinct  data  analysis  procedures  at  its 
disposal;  the  number  continues  to  increase.  No  assumptions  have  been  made  about 
their  frequency  of  use;  one  procedure  has  not  been  declared  most  likely  to  be 
exercised,  a  second  procedure  next  most  likely,  and  so  on,  since  the  base  of  potential 
users  is  so  broad.  For  the  user,  this  means  that  any  procedure  is  a  likely  starting  point, 
as  in  this  session,  the  dialog  of  which  begins  in  figure  2.  In  the  remainder  of  this 
section,  the  conventions  that  boldface  denotes  system  prompts  and  brackets  contain 
user  input  will  be  adopted.  An  occasional  system  response  may  be  italicized  but 
should  not  be  confusing  within  the  context  of  its  appearance. 


Do  you  have  a  sample  > 

(I.  .... 

Xn?  y 

Are  you 

interested  in 

whether 

the  da t 

a  conform  to  a  specified 

distribu 

1 1  on?  n 

Are  you 

interested  in 

the  prob 

abi 1 i ty 

of  occurrence  of  a  particular 

category 

or  event?  y 

Enter  t.h 

e  name  of  the 

c  a  t.egor  y 

of  Hit 

erest  ->  perforation 

Are  the 

n  trials  prod'. 

icing  the 

va  lues 

XI . Xn  independent?  A 

Figure  2.  Beginning  dialog  with  Nonpare. 

The  session  begins  with  a  question  about  the  configuration  of  the  data. 

Do  you  have  a  sample  X,,  ...,  Xn  ?  The  data,  n,  ...,  p,  look  like  X, . Xn;  respond 

[y]es. 

Are  you  interested  in  whether  the  data  conform  to  a  specified  distribution  ? 

Nonpare  is  investigating  a  possible  goodness-of-fit  situation.  A  statistician, 
anticipating  an  approach  to  this  problem,  might  find  a  [y]es  response  is  appropriate 
here.  A  nonstatistician,  for  whom  this  portion  of  the  system  is  designed,  and  who  is 
interested  in  whether  Prfperforation}  >  .80,  should  respond  [n]o,  as  indicated. 

Are  you  interested  in  the  probability  of  occurrence  of  a  particular  category  or 
event  ?  [y]es.  The  user  is  interested  in  the  probability  of  occurrence  of  a  perforation. 


Enter  the  name  of  the  category  of  interest,  [perforation].  Domain-dependent 
terminology  is  being  introduced. 

Are  the  n  trials  producing  the  values  Xlt  ....  Xn  independent  ?  Suppose  the  user  is 
unsure  of  the  technical  implications  of  the  term  "independent."  An  acceptable 
response  is  [What  is  independent]  -  as  shown  in  figure  3. 


Are  the  n  trials  producing  the  values  Xi . Xn  Independent?  what  Is  In 

dependent 

Independence  relates  to  freedom  from  external  influence  or  control-  here , 
the  reference  is  to  measurements  (data)  being  free  to  assume  values 
without  regard  to  other  measurements  that  may  be  matte, 

Are  the  n  trials  producing  the  values  XI,  ,,,,  Xn  Independent?  y 
Does  each  trial  have  the  same  probability  p  of  producing  the  perforation 
?  y 

Are  you  Interested  In  considering  whether  the  probability  of  occurrence 
of  the  perforation  equals  or  is  Bounded  by  some  specified  value  p*?  A 


Figure  3.  A  call  to  the  system  dictionary. 

Independence  relates  to  freedom  from  external  influence  or  control  -  here,  the  reference 
is  to  measurements  (data)  being  free  to  assume  values  without  regard  to  other 
measurements  that  may  be  made. 

This  illustrates  a  dilemma  for  the  subject  area  specialist.  It  may  be  impossible  to 
rigorously  define  a  term  without  reliance  upon  other  terms  that  are  equally  obscure  to 
a  user  with  only  a  modest  statistical  background,  This  is  the  case  here,  where 
independence  is  bound  to  basic  concepts  of  probability  theory.  Nonpare’s  response 
conveys  the  notion,  but  regrettably  not  the  substance,  of  independence.  More  work  is 
needed  here.  For  now,  assume  the  experimenter  has  collected  a  set  of  independent 
data. 

Are  the  n  trials  producing  the  values  Xp  ...,  Xn  Independent  ?  [y]es. 

Does  each  trial  have  the  same  probability  p  of  producing  the  perforation  ?  [y]es. 
Notice  that  Nonpare  is  now  using  language  the  user  provided,  when  it  talks  about 
probability  of  perforation. 

Are  you  interested  in  considering  whether  the  probability  of  occurrence  of  the 
perforation  equals  or  is  bounded  by  some  specified  value  p*  ?  [y]es.  The  user  is 
interested  in  the  inequality  Pr{ perforation}  >  .80.  After  a  [y]es  response,  the  system 
suggests  a  possible  approach,  shown  in  figure  4. 
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The  binomial  test  is  an  appropriate  procedure,  To  execute  the 
binomial  test,  use  the  menu  to  complete  this  statement: 

I  am  interested  in  testing  the  null  hypothesis  that i  The  probability 
of  occurrence  of  the  perforation 


equa l s  some  value  p*| 
does  not  exceed  p* 
is  at.  least  p* 


Figure  4.  A  call  to  the  nonparametric  data  analysis  subsystem. 


The  menu  allows  the  user  to  select  either  a  two-sided  or  one-sided  test  of  hypothesis 
and  is  a  potential  source  of  error.  Beginning  statistics  students,  not  realizing  that  a 
null  (or  empty)  hypothesis  is  chosen  to  be  rejected,  might  mistakenly  choose  is  at  least 
p*  at  this  juncture.  Here  again,  some  level  of  statistical  competence  is  required. 
Selecting  the  hypothesis  does  not  exceed  p*  from  the  menu  using  a  mouse,  the  user 
obtains  for  confirmation  (figure  5)  the  statement: 

I  am  Interested  in  testing  the  null  hypothesis  that:  The  probability  of  occurrence  of  the 
perforation  does  not  exceed  p*. 


I  am  Interested  In  testing  the  null 
occurrence  of  the  perforation  does 

hypothesis  that : 7 he  props 
not  exceed  p* 

bility  of 

Specify  the  sample  size  n  ->  14 

Specify  a  value  for  p*  ->  .80 

Specify  the  number  of  datum  values 

ass  1 gned  t  o  t  he  pe  t  f o rat  i o 

n  -  11 

Figure  5.  Hypothesis  confirmation  and  input  parameter  declaration. 


Specify  the  sample  size  n.  [  14] 
Specify  a  value  for  p*.  (.80] 
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Specify  the  number  of  datum  values  assigned  to  the  perforation.  [11] 

The  first  two  "Specify ... "  commands  determine  the  appropriate  binomial  distribution; 
the  third  determines  the  size  of  the  critical  region  for  the  statistical  procedure,  which 
is  explained  in  figure  7,  following  the  system-generated  histogram  shown  in  figure  6. 
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8  9  10  11  12  13  14 


Figure  6.  Statistical  graph  summary. 

The  histogram  displays  the  probability  of  observing  exactly  n  (n*0 . 14) 

armor  perforations  in  fourteen  shots  if  the  true  (but  unknown)  Pr{perforation}  -  .80. 
A  statistician  will  readily  assimilate  this  graph.  If  the  user  merely  looks  at  it  as  a  plot 
involving  n  rounds  in  which  the  light  gray  region,  corresponding  to  n  >  11,  holds  some 
special  significance,  and  it  provides  some  reassurance  regarding  the  unseen 
computations,  it  will  have  served  its  purpose  here.  Figure  7,  which  appears  on  the 
terminal  simultaneously,  explains  that 

The  critical  level  of  this  test,  corresponding  to  the  light  gray  region,  is  ,69 


This  means  that  if  you  reject  the  hypothesis  ( The  probability  of  occurrence  of  the 
perforation  does  not  exceed  .8)  you  do  so  with  a  .69  probability  of  being  In  error. 


The  critical  leva)  of  this  test,  corresponding  to  the  light  gray  region, 
Is  .69 

This  means  that,  if  you  reject,  the  hypothesis  ( The  prabebi  1  ity  of 
occurrence  of  the  perforation  does  not  exceed  .3)  you  do  so  with  a  ,69 
probability  of  being  In  error, 

Would  you  like  to  run  the  binomial  procedure  again? 


Figure  7.  Explanation  and  interpretation  of  results. 

Since  the  investigation  began  with  the  assumption  (null  hypothesis)  that  the 
Pr {perforation}  <  .80,  the  evidence  collected -eleven  perforations,  three 
nonperforations -is  not  sufficient  to  support  abandonment  of  that  assumption.  A 
probability  of  being  in  error  of  .69  is  more  than  a  reasonable  person  would  be  willing 
to  assume.  And  so,  the  response  to  the  original  question,  Is  the  Pr{perforation}  > 
.80  ?  is  a  qualified  no,  the  qualification  being  expressed  through  invocation  of  the 
critical  level. 

Would  you  like  to  run  the  binomial  procedure  again  7 

At  this  juncture,  an  experimenter  might  well  be  asking  a  number  of  "What  if  ..." 
questions.  "What  if  I  had  been  able  to  afford  three  more  firings?"  or,  "What  if  I  had 
observed  one  more  perforation?"  and  so  on.  A  response  of  [y]es  here  allows  the  user 
to  exercise  the  binomial  procedure  directly,  without  having  to  respond  again  to  all  the 
preliminary  questions.  A  [n]o  response  is  given,  but  this  is  an  excellent  place  to  use 
Nonparc’s  tutorial  capabilities  to  study  the  sensitivity  of  the  binomial  procedure  to 
modification  of  parameter  values  or  slight  changes  in  the  data. 

Are  you  interested  In  determining  an  interval  in  which  the  probability  p  of 
occurrence  of  the  perforation  lies? 

The  foregoing  analysis  suggests  that  an  assertion  that  the  probability  of  perforation 
lies  in  the  interval  (.80,  1]  cannot  be  made.  What  interval  might  be  expected  to 
capture  this  unknown  parameter?  A  response  of  [y]es  causes  this  question  to  be 
answered,  first  graphically,  as  in  figure  8,  and  then  verbally,  as  in  figure  9. 
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Figure  8.  Display  for  a  95$  confidence  interval. 


Figure  8  shows  that  the  Pr {perforation},  whose  estimate  based  on  the 
fourteen  firings  is  p  -  .78,  lies  within  the  interval  [.48,  .94]  with  a  high  level  of 
confidence.  This  interval  is  so  broad  one  can  see  why  the  assertion  that 
Pr{perforation}  >  .80  is  ill-advised.  The  formal  interpretation  of  the  confidence 
interval  is  given  as 

The  probability  of  occurrence  of  the  perforation  is  contained  in  the  interval  [.48, 
.94]  with  an  a  priori  probability  .95 . 


Are  you  interested  in  determining  en  interval  in  which  the  probability  p 
of  occurrence  of  the  perforation  lies?  y 

The  probability  of  occurrence  of  the  perforation  is  contained  in  the 
interval  [,48,. 94]  with  an  a  priori  probability  ,95. 

Would  you  lil*e  a  confidence  level  other  than  .95  '?  n 


Figure  9.  Explanation  and  interpretation  of  the  confidence  interval. 

Would  you  like  n  confidence  level  other  than  .95  ?  [no].  The  95%  confidence  level 
was  prechosen.  A  [y]es  response  allows  the  user  to  control  the  confidence  level.  The 
session  is  terminated  with  a  [n]o  response,  shown  in  figure  9. 

At  the  conclusion  of  the  session  the  inference  engine  displays  a  fact  solution 
tree  for  all  the  intermediate  decisions  leading  to  the  final  conclusion.  Buttoning  with  a 
mouse  any  node  of  the  fact  tree  produces  the  logic  leading  to  that  location.  In  figure 
10,  factll  was  buttoned,  and  the  corresponding  trace  is  displayed  beneath  the  fact 
tree.  These  are  features  of  the  inference  engine  rather  than  Nonpare,  but  they  are 
valuable  as  diagnostics  to  the  developer  and  provide  some  measure  of  reassurance  to 
the  user. 
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Figure  IQ.  Session 


4.  Conclusions 

Nonparc,  a  consultation  system  for  analysis  of  data  using  nonparametric 
statistical  procedures,  has  been  described;  and  most  of  its  operational  features  have 
been  illustrated.  The  essence  of  the  system  is  the  rule-based  interface  with 
accompanying  software  for  data  analysis  and  the  interpretation  of  the  ensuing 
computations.  Nonpart  is  under  active  development,  but  its  feasibility  as  an 
operational  system  has  been  established.  Enlargement  of  the  rule-base  and  the 
addition  of  more  statistical  procedures  is  clearly  indicated  before  it  can  approach  its 
potential.  Not  surprisingly,  tangential  problems  in  basic  research  have  been  spawned 
by  this  effort.  A  first  release  is  planned  for  1989  for  field  testing. 
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abstract.  The  parameters  which  maximize  the  log-likelihood  function 
of  the  Gumbel  distribution  were  estimated  by  two  different  methods.  A 
derivative  approach  was  used,  which  calculated  the  intersection  of  the 
zeros  of  the  implicit  functions  obtained  from  the  derivatives  of  the 
log- likelihood  function.  A  direct  maximization  was  also  performed. 

Both  methods  yielded  positive  results. 
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EXPERIMENTAL  DESIGN  AND  OPTIMIZATION  OF  BLACK  CHROME 
SOLAR  SELECTIVE  COATINGS 

I.  J.  Hall  and  R.  B.  Pettit 
Sandia  National  Laboratories 
Albuquerque,  NM  87185 

ABSTRACT .  Some  years  ago  Sandia  Laboratories  was  given  the 
task  of  investigating  selective  coatings  for  solar  applications. 
Early  experimental  results,  which  were  based  on  one  variable  at 
the  time  experiments,  produced  acceptable  coatings  in  the 
laboratory.  However,  when  full  scaled  parts  were  coated  by 
commercial  electroplaters,  the  coatings  quickly  degraded  when 
heated  in  air.  At  this  point  a  systematic  approach  using  a 
fractional  factorial  design  was  used  to  determine  both  the 
effects  and  interactions  between  several  variables,  including  the 
bath  composition (four  variables),  current  density,  plating  time, 
substrate,  and  bath  temperature.  Response  surface  for  the 
optioal  properties  of  the  coatings  were  constructed  for  both  the 
as-plated  and  the  thermally  aged  samples.  These  response 
surfaces  were  then  used  to  specify  ranges  for  the  bath 
compositions,  and  other  plating  parameters,  that  provided 
coatings  with  optimum  thermal  stability.  With  proper  control  of 
the  plating  variables,  selective  coatings  were  obtained  that 
should  maintain  high  solar  absorptance  values  during  years  of 
operational  at  300®C  in  air. 

1.  INTRODUCTION.  Two  variables  are  of  interest  to 
selective  coating  investigators,  namely,  absorptance  (a)  and 
emittance  (c) .  Good  selective  coatings  have  high  a's  and  low 
«'s.  In  our  investigations  we  concentrated  on  making  a  as  large 
as  possible  and  settling  for  the  corresponding  «  if  it  was  not 
"too  big".  The  independent  variables  that  affected  a  and  e 
divided  themselves  into  two  groups  (bath  variables  and  plating 
variables)  in  such  a  way  that  a  split-plot  experimental  design 
would  have  been  appropriate.  The  bath  variables  would  have  been 
associated  with  the  whole  plots  and  the  plating  variables  with 
the  sub-plots.  The  bath  variables  were  chromic  acid,  trivalent 
chromium,  addition  agent  and  iron,  and  the  plating  variables  were 
plating  time,  current  density,  bath  temperature,  bath  agitation 
and  substrate.  For  a  specified  combination  of  bath  variables  a 
entire  set  of  experiments  were  possible  for  the  plating  variables 
as  in  a  split-plot  design.  Because  of  many  constraints  we  did  run 
the  experiment  as  a  split-plot  design.  The  dependent  variable 
readings  (  a's  )  were  obtained  by  coating  a  substrate  and  then 
measuring  the  absorptance  with  a  Beckman  Model  DK_2A 
spectroref lactometer.  Readings  were  obtained  for  the  substrate 
both  as-plated  and  as-aged.  The  as-aged  readings  were  obtained 
after  the  specimens  were  heated  in  a  450°c  furnace  for  40  hours 
while  the  as-plated  readings  were  taken  before  they  were 
subjected  to  any  extreme  environments.  The  aged  readings  were 
the  most  important  because  we  were  concerned  about  the  thermal 
stability  of  the  coatings,  i.e.  would  coatings  not  degrade  at 
high  temperature  for  extended  time  periods.  The  experimentation 
was  done  in  three  phases  that  are  briefly  described  below. 
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2.  Experimentation.  Based  on  previous  experience,  we 
decided  that  the  bath  variables  were  most  important  and  thus  we 
concentrated  most  of  our  efforts  on  investigating  these 
variables.  The  plating  variables  were  set  at  nominal  values.  We 
used  standard  response  surface  methodology  to  guide  us  in  the 
experimentation.  (See  Box,  Hunter,  and  Hunter,  "Statistics  for 
Experimenters",  Chapter  15,  1978)  The  first  phase  consisted  of 
running  a  1/2  replicate  of  a  24  factorial  experiment  on  the  four 
bath  variables.  The  experimentation  was  done  in  a  rather  limited 
range  of  the  factor  space.  The  results  of  this  experiment  were 
used  to  determine  a  path  of  steepest  ascent  (Phase  two) .  Three 
more  experiments  were  done  along  this  line  of  steepest  ascent. 
These  experiments  would  normally  indicate  a  region  in  the  bath 
variable  space  that  would  produce  larger  a  values.  In  our  case 
however  all  the  coatings  turned  gray  after  a  short  time  in  the 
furnace  -  a  highly  undesirable  result.  The  most  valuable 
information  from  these  three  bath  experiments  was  that  a  "cliff" 
existed  in  the  response  surface.  Because  of  time  limitations  we 
did  not  repeat  the  experiments  along  the  steepest  ascent  line. 
Based  on  a  combination  of  engineering  judgement  and  factorial 
design  methodology,  several  more  baths  were  mixed  and  the  a' a 
measured  on  the  coated  substrates  (Phase  three) .  A  total  of 
eighteen  baths  ware  mixed  and  the  results  from  these  baths  were 
used  to  estimate  a  quadratic  surface  -  i.e.  a  was  written  as  a 
function  of  a  second  degree  polynomial  in  the  four  bath  variables 
and  the  variable  coefficients  were  estimated  using  a  backward 
stepwise  statistical  package.  The  final  regression  equation  had 
11  terms  including  the  constant  term  with  an  Rz  -  0.96.  Several 
graphs  were  drawn  based  on  this  equation  that  allowed  us  to  map 
out  an  acceptable  region  in  the  bath  variable  space.  This  space 
was  very  near  the  "cliff"  in  the  response  surface.  A  limited 
number  of  experiments  also  were  done  involving  the  plating 
variables  for  a  fixed  bath.  Based  on  these  experiments  we  were 
able  to  specify  ranges  for  the  plating  variables  as  well. 

3.  Summary.  Using  response  surface  methodology  we  were 
able  to  determine  the  variables  and  the  range  of  variables  that 
produced  stable  selective  coatings.  The  procedures  developed  in 
the  laboratory  were  subsequently  implemented  in  a  production 
environment  with  excellent  results.  The  close  interaction 
between  the  statistician  and  the  experimenter  led  to  a 
satisfactory  solution  with  a  rather  limited  number  of 
experiments . 
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And  Engineering  Center 
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ABSTRACT 


Field  evaluations  have  determined  that  camouflage  patterns  reduce  detectability  ranges  for 
uniforms  and  vehicles  in  woodland  environments.  This  study  identified  the  effects  of  three  pat¬ 
terned  and  two  monotoned  Five-Soldier  Crew  Tents  using  detection  ranges  and  number  of  false 
detections  as  determined  by  ground  observers.  The  distance  of  correct  detections  were  recorded 
along  with  the  number  of  false  detections.  An  analysis  of  variance  for  the  detection  ranges  and 
number  of  false  detections  was  performed.  The  Duncan’s  Multiple-Range  Test  was  used  to 
determine  significant  differences  (a  «  0.05)  in  groups  of  tents.  From  this  data,  it  was  deter¬ 
mined  that  the  three  patterned  Five-Soldier  Crew  Tents  were  more  difficult  to  detect  than  the 
two  monotone  tents. 

1.0  SECTION  I  -  INTRODUCTION 

Several  years  ago,  the  U.S.  Army  decided  that  camouflage  patterns  have  a  definite  ad¬ 
vantage  when  used  on  uniforms  and  vehicles  in  woodland  environments.  This  had  led  to  a  similar 
consideration  for  tents,  since  the  current  U.S.  Army  tents  are  solid  (i.e.,  monotone)  color.  Tents 
present  a  large,  relatively  smooth  form,  making  them  conspicuous  targets.  The  use  of  patterns 
to  break  up  this  signature  could  increase  camouflage  effectiveness.  However,  before  such  a 
judgement  could  be  made,  a  field  tent  was  planned  to  determine  the  relative  merits  of  various 
patterns  versus  monotones  in  a  woodland  background.  The  Natick  RD&E  Center  fabricated 
three  patterned  tents  and  two  monotone  tents  for  evaluation.  In  consultation  with  Belvoir,  the 
patterned  tents  were  fabricated  in  the  standard  four-color  uniform  pattern,  one  in  the  standard 
pattern  size  and  the  other  two  in  progressively  larger  expanded  patterns.  The  two  monotone 
tents  were  in  colors  Forest  Green  and  Green  483  (483  being  the  textile  equivalent  of  paint  color 
Green  383).  A  tost  plan’^  was  developed  by  Belvoir  at  the  request  and  funding  of  Natick,  and 
the  field  test  was  conducted  by  Belvoir  at  Ft.  Devens,  Massachusetts,  in  the  summer  of  1987. 
This  report  describes  the  test  and  its  results. 
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2.0  SECTION  II  •  EXPERIMENTAL,  DESIGN 


2.1  Test  Targets 

Five,  Five-Soldier  Crew  Teats  were  supplied  by  Natick  for  this  study  ia  the  following  pat¬ 
terns  and  colors: 

•  Tent  A  •  Standard  size  four-color  uniform  pattern  repeated  every  27.25  inches 

•  Tent  B  •  Forest  Green 

a  Tent  C  -  Expanded  four-color  uniform  pattern  repeated  every  36  inches 
a  Tent  D  •  Expanded  four-color  uniform  pattern  repeated  every  50  inches 
a  Tent  E  •  Green  483 

2.2  Test  Sites 

The  study  was  conducted  at  the  Turner  Drop  Zone,  Ft.  Devens,  Massachusetts,  a  large 
cleared  tract  of  land  surrounded  by  a  mix  of  coniferous  and  deciduous  trees  resembling  a  central 
European  forest  background.  Two  test  sites  were  selected.  Site  #1  was  located  on  the  western 
end  of  the  drop  zone,  so  that  the  morning  sun  shone  directly  upon  the  test  tent.  Site  #2  was 
located  on  the  eastern  edge  of  the  drop  zone,  so  that  the  afternoon  sun  shone  directly  upon  the 
test  tent.  An  observation  path,  starting  at  the  opposite  end  of  the  drop  zone  from  the  test  tent 
location,  was  laid  out  for  each  site.  Each  path  followed  zig-zag,  random  length  directions  toward 
its  test  site,  and  afforded  a  continuous  line  of  sight  to  its  respective  test  tent  location.  The 
paths  were  within  a  30°  to  40°  cone  from  the  target  tents,  and  were  surveyed  and  marked  at  ap¬ 
proximately  50-meter  intervals  using  random  letter  markers.  For  Site  #2,  the  distance  between 
markers  after  the  first  15  markers  was  about  25  meters  along  the  path.  A  night  evaluation  in¬ 
volving  other  camouflage  targets  led  to  this  procedural  change.  The  markers  and  distances  from 
the  tents  are  shown  in  Tables  1  and  2, 


Table  1 


Distances  of  Markers  to  Tents  for  Site  #1 


ALPHABET 

DISTANCE  IN 

ALPHABET 

DISTANCE  IN 

MARKER 

METERS  ALONG 

MARKER 

METERS  ALONG 

PATH  FROM 

PATH  FROM 

STARTING  POINT 

STARTING  POINT 

TO  TENT 

TO  TENT 

S 

1,182.64 

S' 

464.78 

Y 

1,128.57 

Y' 

446.74 

Q 

1,094.00 

Q' 

428.17 

L 

1,049.93 

L' 

413.48 

F 

1,008.07 

F' 

398.46 

P 

978.31 

P ' 

383.34 

E 

947.02 

E 

364.34 

K 

902.76 

K' 

348.27 

A 

858.10 

A' 

334.46 

T 

817.81 

T' 

322.69 

V 

778.91 

V' 

306.59 

B 

750.15 

B' 

289.59 

M 

709.76 

'  M' 

281.60 

U 

674.87 

U' 

269.08 

H 

702.65 

H' 

253.16 

Z 

677.99 

Z' 

235.50 

R 

648.46 

R' 

217.81 

N 

613.35 

N' 

199.60 

X 

602.56 

X' 

178.93 

1 

594.57 

1* 

158.76 

D 

578.05 

D' 

141.15 

C 

561.16 

C' 

120.05 

0 

541 .70 

O  ' 

102.34 

J 

525.33 

J' 

85.37 

G 

505.62 

G  ' 

62.81 

W 

483.64 

W* 

41.84 
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Table  2 


Distances  of  Markers  to  Tents  for  Site  #2 


ALPHABET 

0I8TANCE  IN 

ALPHABET 

DISTANCE  IN 

MARKER 

METERS  ALONG 

MARKER 

METERS  ALONG 

PATH  FROM 

PATH  FROM 

STARTING  POINT 

STARTING  POINT 

TO  TENT 

TO  TENT 

F 

1 ,205.36 

A 

653.34 

W 

1,168.83 

Z 

613.20 

U 

1,130.58 

E 

574.09 

Q 

1,086.03 

P 

540.30 

C 

1,048.10 

H 

513.10 

R 

1,006.15 

K 

496.46 

V 

982.00 

S 

475.57 

0 

974.13 

F' 

459.10 

M 

942.37 

W' 

417.71 

I 

901.58 

U' 

379.40 

B 

869.75 

O' 

338.25 

J 

858.01 

C' 

296.90 

L 

851.64 

R' 

278.53 

X 

841.26 

V' 

258.20 

G 

803.96 

O' 

220.73 

D 

764.09 

1  • 

180.87 

Y 

723.46 

B' 

143.94 

T 

695.32 

J' 

111.00 

N 

873.60 

L' 

69.78 

23  Test  Subjects 

A  total  of  1S3  enlisted  soldiers  from  Ft.  Devens  served  as  ground  observers.  All  person¬ 
nel  bad  at  least  20/30  corrected  vision  and  normal  color  perception.  A  minimum  of  30  observers 
were  used  for  each  test  tent,  about  evenly  split  between  test  sites.  Each  observer  was  used  only 
once. 

2.4  Data  Generation 

The  test  procedure  was  to  determine  the  detection  distances  of  the  five  tents  involved  by 
searching  for  them  while  traveling  along  the  predetermined  measured  paths.  Each  ground  ob¬ 
server  started  at  the  beginning  of  the  observation  path,  i.e.,  marker  S  for  Site  #1  and  marker 
F  for  Site  #2.  The  observer  rode  in  the  back  of  an  open  5/4-ton  truck  accompanied  by  a  data 
collector.  The  truck  traveled  down  the  observation  path  at  a  very  slow  speed,  about  3-5  mph. 
The  observer  was  instructed  to  look  for  military  targets  in  all  directions  except  directly  to  his 
rear.  When  a  possible  target  was  detected,  the  observer  informed  the  data  collector  and  pointed 


to  tbe  target.  The  truck  was  immediately  stopped,  and  the  data  collector  sighted  the  apparent 
target.  If  the  sighting  was  correct,  i.e.,  the  Five-Soldier  Crew  Tent,  the  data  collector  recorded 
the  alphabetical  marker  nearest  the  truck.  If  the  detection  was  not  correct,  the  false  detection 
was  recorded,  and  the  data  collector  informed  the  observer  to  continue  looking.  The  truck 
proceeded  down  the  observation  path.  This  search  process  was  repeated  until  the  correct  tar¬ 
get  (tent)  was  located. 

The  tents  were  rotated  between  the  two  test  sites  on  a  daily  basis,  until  all  tents  had  been 
observed  by  at  least  15  observers  at  each  site.  (This  number  of  observers  allows  the  use  of 
parametric  statistics  which  have  a  good  opportunity  to  yield  absolute  conclusions).  Their  orien¬ 
tations  with  respect  to  the  sun  were  kept  constant  at  both  test  sites.  The  Five-Soldier  Crew 
Tent  was  positioned  so  that  a  full  side  was  facing  the  direction  of  observer  approach. 

3.0  SECTION  III-RESULTS 
3.1  Range  of  Detection 

Tables  3,  4,  and  5  show  the  detection  data  for  the  Five-Soldier  Crew  Tents.  Table  3  gives 
the  mean  detection  range  in  meters  for  each  tent,  and  its  associated  95  percent  confidence  in¬ 
terval.  Table  4  shows  the  analysis  of  variance^  performed  upon  the  data  of  Table  3  to  deter¬ 
mine  if  there  were  significant  differences  in  the  detection  ranges,  i.e.,  if  pattern  and  color  had 
an  effect  upon  detection  range.  Table  5  indicates  which  tent  patterns  and  solid  colors  differed 
significantly  from  each  other  in  this  respect.  Figure  1  is  a  graphic  display  of  the  detection  ran¬ 
ges  of  Table  3. 

Table  3 

Mean  Detection  Ranges  (Meters)  and  95  Percent 
Confidence  Intervals. 

96  PERCENT  CONFIDENCE 
STANDARD  INTERVAL 


TENT 

N 

MEAN 

ERROR 

LOWER  LIMIT 

UPPER  LIMIT 

A 

31 

327.54 

127.75 

280.68 

374.40 

B 

30 

427.71 

173,74 

362.83 

492.58 

C 

32 

351.17 

129.42 

304.51 

397.83 

D 

30 

387.12 

181,79 

326.76 

447.59 

E 

30 

674.88 

214,94 

594.62 

755.14 
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Table  4 


Analysis  of  Variance  for  Tent  Detection 
Across  Five  Levels  of  Color  Variation 


DEGREES 

OF 

SOURCE 

FREEDOM 

SUM  OF  SQUARES 

MEAN  SQUARE 

F-TEST 

SIQ  LEVEL 

TENT  COLOR 

4 

2,377,907  968 

594,476.9927 

22.0083 

0.00* 

ERROR 

148 

3,983,214.260 

26,913.6099 

TOTAL 

152 

6,361,122.228 

*Significant  at  a  less  than  0.001  level. 


Table  4  indicates  that  there  are  significant  differences  in  the  ability  of  the  ground  observers 
to  detect  the  Five-Soldier  Crew  Tents  in  different  four-color  patterns  and  solid  colors 
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Figure  1.  Mean  Ranges  of  Detection  and  95  Percent 
Confidence  Intervals  for  Five-Soldier  Crew  Tents 
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Table  5 


Duncan’s  Multiple-Range  Test  (Range  of  Detection) 


SUBSET  1  SUBSET  2  SUBSET  3 


GROUP 

MEAN 

GROUP 

MEAN 

GROUP 

MEAN 

A 

327.54 

C 

351.17 

E 

874.88 

C 

351.17 

D 

387.12 

D 

387.12 

B 

427.71 

The  harmonic  mean  group  size  is  30.58.  The  subsets  are  significant  at  a  ■  0.05 

The  Duncan’s  Multiple-Range  test  separates  a  set  of  significantly  different  means  into  sub¬ 
sets  of  homogeneous  means.  One  of  the  assumptions  is  that  each  random  sample  is  of  equal 
size.  Since  this  was  not  true,  the  harmonic  mean  of  the  group  was  used  as  the  group  size.  As 
seen  above,  the  range  of  detection  was  the  shortest  for  tents  A,  C,  and  D  and  these  tents  do 
not  differ  significantly  from  each  other  (a  ■  0.05).  Tent  E  had  the  longest  mean  range  of  detec¬ 
tion  and  is  significantly  (a  -  0.05)  different  from  the  other  4  tents  in  this  respect. 

3 1  False  Detections 

The  number  of  false  detections  is  defined  as  the  number  of  times  a  target  other  than  the 
test  target  is  detected  by  an  observer.  In  this  study  such  detections  are  rocks,  trees,  shadows, 
etc.  These  detections,  as  a  rule,  are  a  function  of  how  hard  it  is  to  detect  the  test  target.  The 
more  difficult  the  detection  task,  the  greater  the  number  of  false  detections.  Tables  6,  7,  and 
8  show  the  false  detection  data.  Table  6  gives  the  mean  false  detection  value,  and  its  associated 
95  percent  confidence  interval,  for  each  of  the  Five-Soldier  Crew  Tents.  Table  7  contains  the 
analysis  of  variance  performed  upon  the  data  of  Table  6  to  determine  if  there  were  significant 
differences  in  the  rate  of  false  detections.  Table  8  indicates  which  tent  patterns  and  colors  had 
significant  rates  of  false  detection. 


Table  6 

Mean  False  Detection  Rates  and  95  Percent  Confidence  Intervals 

95  Percent  Confidence 
Standard  Interval 


Tent 

N 

Mean 

Error 

Lower  Limit 

Upper 

A 

31 

4.87 

3.27 

3,67 

6,07 

B 

30 

3.53 

2.53 

2.59 

4.48 

C 

32 

3.38 

1.96 

2.67 

4.08 

D 

30 

3.87 

2.76 

2.83 

4.90 

E 

30 

2.50 

1.91 

1.79 

3.21 
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Tabic  7 


Analysis  of  Variance  for  Rates  of  False 
Detection  across  Five  Levels  of  Color  Variance 

DEGREES 


SOURCE 

ur 

FREEDOM 

SUM  OF  SQUARES 

MEAN  SQUARE 

F-TEST 

LEVEL 

TENT  COLOR 

4 

90.086 

22.521 

3.50 

0.009 

ERROR 

148 

953.417 

6.44 

TOTAL 

152 

1043.503 

Significant  at  less  than  0.01  level. 

Table  7  indicates  that  there  are  significant  differences  in  the  rates  of  false  detection  for 
the  Five-Soldier  Crew  Tents. 
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Figure  2.  Mean  Rates  of  False  Detection  and  95  Percent 
Confidence  Intervals  For  Five-Soldier  Crew  Tents 
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Table  8 


Duncan’s  Multiple-Range  Test 
(Rates  of  False  Detection) 


SUBSET  1  SUBSET  2 


GROUP 

MEAN 

GROUP 

MEAN 

E 

2.50 

B 

3.53 

C 

3.38 

D 

3.87 

B 

3.53 

A 

4.87 

D 

3.87 

Harmonic  mean  group  size  is  30.58. 

The  rates  of  false  detection  for  tent  groups  E,  C,  B,  and  D,  and  B,  D,  and  A  were  not  sig¬ 
nificantly  different  (a  -  0.05).  However  subset  1  is  significantly  different  from  subset  2. 


4.0  SECTION  IV -DISCUSSION 

The  Duncan's  Multiple-Range  Test  (Table  5)  shows  that  the  group  of  Five-Soldier  Crew 
Tents  A,  C,  and  D  had  the  shortest  detection  range.  Tent  A  is  the  standard  size  woodland 
uniform  four-color  pattern,  while  Tents  C  and  D  are  expansions  of  this  pattern.  The  pattern  at 
Tent  A  is  repeated  every  27.25  inches,  the  pattern  for  Tent  C  is  repeated  every  36  inches,  and 
the  pattern  for  Tent  D  is  repeated  every  50  inches.  Tents  C,  D,  and  B  are  significantly  different 
from  each  other.  Tent  B  is  solid  color,  Forest  Green,  Tent  E,  which  is  not  solid  color  Green 
483,  had  the  longest  mean  detection  range  (674.89  meters),  and  this  is  significantly  (a  »  0.05) 
longer  than  any  of  the  other  means  for  the  Five-Soldier  Crew  Tents.  Thus,  it  can  be  concluded 
that  the  patterned  tents  are  harder  to  detect  from  ground  observation,  but  that  the  pattern 
should  not  be  expanded  beyond  the  repeat  of  every  36  in<  hes.  The  human  eye  is  probably  resolv¬ 
ing  the  larger  pattern  repeated  every  50  inches  as  beii^  different  from  the  tree  and  bush  back¬ 
ground  (the  color  brown,  in  particular,  becomes  distinguishable  from  the  woodland  background 
when  overexpanded). 

When  working  with  detection  ranges,  the  question  of  Held  data  stability  is  always  paramount 
to  the  amount  of  weight  that  can  be  given  to  the  test  conclusions.  One  of  the  best  methods  to 
determine  data  stability  is  through  a  test-retest  procedure.  Field  studies  are  very  expensive  and 
time  consuming,  so  this  data  is  very  rare.  Wc  do  have  such  an  opportunity  to  examine  this  type 
of  data  for  the  Turner  Drop  Zone.  A  ground  evaluation  of  camouflage  nets  was  conducted  in 
the  summers  of  19853/  and  19874/.  The  net  sites  and  test  procedures  were  identical  to  the  sites 
and  test  procedures  in  which  the  Five-Soldier  Crew  Tents  were  evaluated.  In  both  net  studies, 
the  standard  camouflage  net  was  evaluated.  In  1985  this  net  had  a  mean  detection  range  of 
411.75  meters,  while  in  1987  the  mean  detection  range  was  414.41  meters.  This  difference  in 
mean  detection  range  is  only  2.66  meters.  From  these  results,  it  is  inferred  that  the  mean  detec- 
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t ion  ranges  for  the  Five-Soldier  Crew  Tents  are  stable,  and  solid  conclusions  about  their 
camouflage  effectiveness  can  be  made. 

The  analysis  of  false  detections  seen  in  Table  8  and  Figure  2  also  lends  credence  to  the 
belief  that  the  Five-Soldier  Crew  Tent  A  had  the  best  performance  as  to  camouflage  effective¬ 
ness,  with  Tent  £  the  worst  performance.  In  the  following  discussion  of  false  detections  in  Sec¬ 
tion  3.2,  it  would  be  expected  that  Tent  A,  being  the  hardest  to  find,  would  have  the  most  false 
detections,  and  Tent  E  the  least  number  of  false  detections.  This  is  exactly  what  occurred,  with 
Tent  A  having  a  mean  false  detection  rate  of  4.87,  and  Tent  E  a  mean  false  detection  rate  of 
2.30.  Duncan’s  Multipie-Range  Test  (Table  8)  shows  that  the  two  rates  of  false  detection  dif¬ 
fer  significantly  ( a  *  0,05)  from  each  other.  The  false  detection  rates  of  tents  B,  C,  and  D  are 
not  in  the  expected  ordinal  position.  The  expected  order,  based  upon  mean  range  of  detection, 
would  be  B,  D,  and  C,  while  the  true  order  of  rates  of  false  detection  is  C,  B,  and  D.  However, 
a  check  of  Tables  5  and  8  shows  that  these  tents  are  not  significantly  different  from  each  other 
either  for  range  of  detection  or  for  rate  of  false  detection.  Thus,  from  a  statistical  view,  these 
three  tents  are  considered  to  have  the  same  ordinai  position. 

5.0  SECTION  V-SUMMARY  AND  CONCLUSIONS 

Five,  Five-Soldier  Crew  Tents  were  evaluated  by  ground  observers  to  determine  their 
camouflage  effectiveness  as  measured  by  the  mean  detection  range  and  the  mean  rate  of  false 
detection.  These  tents  were  in  the  following  four-color  camouflage  patterns  and  solid  colors: 

•  Tent  A  -  Standard  size  four-color  uniform  pattern  repeated  every  27.25  inches. 

•  Tent  B  -  Forest  Green 

•  Tent  C  •  Expanded  four-color  uniform  pattern  repeated  every  36  Inches 

•  Tent  D  •  Expanded  four-color  uniform  pattern  repeated  every  50  inches 

•  Tent  E  -  Green  483 


A  minimum  of  30  ground  observers  per  Five-Soldier  Crew  Tent  were  driven  toward  each  of  two 
sites  on  marked  observation  trails  in  the  back  of  an  open  5/4-ton  truck.  The  observers  were 
looking  for  military  targets,  and  they  informed  the  data  collector  when  they  thought  they  saw 
one.  If  the  detection  was  correct,  the  closest  alphabetic  ground  marker  to  the  truck  was  recorded. 
From  this  letter,  the  distance  to  the  tent  from  the  truck  was  determined.  If  the  detection  was 
not  correct,  i.e.,  false  detection,  it  was  noted  on  the  data  sheet.  The  ground  observer  then  con¬ 
tinued  the  search,  with  the  truck  traveling  down  the  observation  path  until  the  test  target  was 
seen.  An  analysis  of  the  resulting  data  provided  the  following  conclusions: 

A,  Five-Soldier  Crew  Tent  A  was  the  most  camouflage  effective,  with  the  lowest  mean 
range  of  detection  and  highest  rate  of  false  detections. 

B,  Four-color  pattern  Five-Soldier  Crew  Tents  are  more  camouflage  effective  than  solid 
colors. 


198 


C.  The  expanded  four-color  pattern,  repeated  every  50  inches,  is  too  large  to  be  effective 
in  denying  detection.  (The  color  brown  becomes  distinguishable  from  the  woodland  background 
when  overexpanded). 

D.  The  solid  colors  Green  483  and  standard  Forest  Green  should  not  be  used. 

E.  The  mean  range  of  detection  data  appears  to  be  very  stable.  A  test-retest  field  study 
using  identical  sites  and  test  procedures  in  the  summers  of  1985  and  1937  involving  the  stand¬ 
ard  camouflage  net  yielded  mean  detection  ranges  of  411.75  and  414.41  meters  respectively. 
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AN  EXAMPLE  OF  CHAIN  SAMPLING  AS  USED  IN  ACCEPTANCE  TESTING 
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ABSTRACT 

The  Probability  and  Statistics  Branch  of  the  Ballistic  Research  Laboratory  was  asked  to 
develop  a  procedure  of  acceptance  testing  for  armor  packages.  Because  the  available  sample 
sizes  were  extremely  small,  we  were  unable  to  identify  a  sampling  plan  directly  applicable  to 
this  problem.  Accordingly,  we  have  devised  a  new  procedure  by  adapting  an  existing  tech¬ 
nique,  known  as  chain  sampling,  to  both  the  attribute  portion  (structural  integrity)  and  the 
variable  portion  (penetration  depth)  of  the  acceptance  testing  process.  Operating  charac¬ 
teristic  curves  and  power  curves  are  presented  for  this  procedure,  and  suggestions  are  made 
concerning  the  simultaneous  use  of  quality  control  charts. 
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I.  INTRODUCTION 


In  most  cases  a  consumer’s  decision  concerning  whether  or  not  to  accept  a  manufac¬ 
tured  product  is  based  on  an  examination  of  a  sample  from  that  product.  When  General 
Mills  introduces  a  new  pre-sweetened  breakfast  cereal,  they  spend  millions  of  dollars  in 
advertisement  costs  with  the  hope  that  the  consumer  will  sample  it.  Here,  the  consumer  con¬ 
siders  the  entire  supply  of  this  new  cereal  as  a  single  manufactured  lot,  to  be  accepted  or 
rejected.  Product  acceptance,  in  this  case,  corresponds  to  the  consumer  purchasing  more 
boxes  of  the  new  cereal. 

This  is  merely  an  everyday  example  of  what  is  known  as  acceptance  sampling,  that  is, 
various  techniques  which  allow  for  discrimination  between  an  acceptable  product  and  an 
unacceptable  one.  Sampling  may  be  based  on  an  attribute  criterion,  a  variable  criterion,  or 
some  combination  of  these.  In  our  example  the  consumer  may  judge  the  sweetness  of  the 
cereal  as  satisfactory  or  excessive  (attribute),  or  he  may  measure  the  time  in  milk  before  the 
cereal  becomes  soggy  (variable).  Sampling  by  attributes  is  a  dichotomous  situation  in  that, 
based  on  a  particular  attribute,  each  item  is  either  defective  or  non-defective;  rejection  occurs 
if  there  is  a  high  percentage  of  defectives  in  the  sample.  Sampling  by  variables  establishes  an 
acceptable  level  of  a  particular  variable,  and  rejection  occurs  if  its  sample  value  crosses  the 
acceptable  threshold.  Of  course,  in  our  example  of  a  box  of  cereal,  the  sample  size  was  one. 
Generally,  this  will  not  be  the  case;  but  occasionally,  for  one  reason  or  another,  the  consumer 
is  forced  to  make  a  decision  based  upon  a  very  small  sample  size. 

Because  decisions  are  made  from  samples,  there  is  some  risk  of  error,  either  the  error  of 
accepting  a  bad  product  or  the  error  of  rejecting  a  good  product.  The  amount  of  protection 
desired  against  such  risks  can  be  specified.  The  Acceptable  Process  Level  (APL)  is  a  high- 
quality  level  that  should  be  accepted  100(l-a)%  of  the  time;  a  is  thus  defined  to  be  the 
producer’s  risk.  The  Rejectable  Process  Level  (RPL)  is  a  low-quality  level  that  should  be 
accepted  only  100 (0)%  of  the  time;  0  is  thus  defined  to  be  the  consumer’s  risk.  Unfor¬ 
tunately,  these  error  factors  vary  inversely;  that  is,  as  the  consumer’s  risk  grows,  the 
producer’s  risk  diminishes  and  vice  versa.  The  Operating  Characteristic  (OC)  curve  is  an 
important  part  of  any  acceptance  sampling  plan,  since  it  provides  a  graphical  display  of  the 
probability  of  accepting  a  product  versus  the  value  of  the  particular  parameter  being 
inspected.  The  OC  curve  is  a  function  of  APL,  RPL,  a,  and  /?,  as  well  as  sample  size.  Given  a 
particular  acceptance  sampling  plan,  the  OC  curve  depicts  the  associated  error  risks  and 
demonstrates  the  relationship  among  all  of  the  variables. 

The  US  Army  Ballistic  Research  Laboratory  (BRL)  has  developed  acceptance  sampling 
plans  for  armor  packages.  These  plans  were  briefed  to  the  Project  Manager  M1A1  on  14 
April  1988  at  Aberdeen  Proving  Ground,  Maryland.  Their  general  structures  were  accepted 
with  the  guidance  that  the  processes  would  be  officially  adopted  pending  some  refinements. 
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Ii.  CHAIN  SAMPLING 


Numerous  sampling  techniques  exist,  each  with  special  properties  that  make  it  applica¬ 
ble  to  particular  situations.  Sampling  plans  reviewed  in  the  literature  required  sample  sizes 
much  larger  than  those  feasible  for  armor  testing.  In  our  case  extremely  small  sample  sizes 
were  warranted  due  to  the  expense  of  both  the  armor  and  the  testing  procedure,  augmented 
by  the  destructive  nature  of  the  test  itself.  Accordingly,  we  have  devised  a  new  procedure  by 
adapting  an  existing  technique,  chain  sampling,  for  use  in  this  project. 

Chain  sampling  is  particularly  appropriate  for  small  samples  because  it  uses  information 
over  the  past  histoiy  of  production  lots.  Even  with  small  samples,  it  is  possible  to  accept  a 
marginal  lot  provided  that  a  given  number  of  lots  immediately  preceding  (i.e.,  the  chain)  were 
acceptable.  When  a  consumer  uses  an  expendable  product  such  as  the  breakfast  cereal  in  our 
previous  example,  he  utilizes  chain  sampling  in  his  decision  of  whether  or  not  to  subsequently 
purchase  the  same  product.  If  the  first  or  second  box  he  buys  is  unacceptable,  he  will  prob¬ 
ably  discard  the  product  forever.  However,  if  the  tenth  box  is  unacceptable,  he  might  con¬ 
tinue  with  one  more  purchase  of  the  same  cereal  taking  into  consideration  its  past  history  of 
nine  boxes  of  acceptable  quality. 

An  advantage  of  chain  sampling  is  its  automatic  incorporation  of  reduced  or  tightened 
inspection  procedures  when  applicable.  That  is,  as  quality  remains  acceptable  over  a  period 
of  time  and  our  confidence  grows,  the  sample  size  is  reduced  (or,  more  accurately,  samples 
are  taken  less  frequently).  If  quality  becomes  marginal,  inspection  is  tightened  by  taking  sam¬ 
ples  more  frequently.  When  quality  diminishes  to  the  point  where  a  production  lot  must  be 
rejected,  the  production  process  is  stopped  and  necessary  adjustments  and  corrections  are 
made.  At  that  point  a  new  chain  begins  and  continues  as  before. 

Certain  assumptions  must  be  made  before  chain  sampling  is  considered  as  a  sampling 
technique.  In  particular,  production  should  be  a  steady,  continuous  process  in  which  lots  are 
tested  in  the  order  of  their  production.  Also,  there  should  be  confidence  in  the  supplier  to 
the  extent  that  lots  are  expected  to  be  of  essentially  the  same  quality.  Generally,  a  fixed  sam¬ 
ple  size  will  be  maintained  with  the  investigator  taking  more  or  fewer  samples  as  tightened  or 
reduced  inspection  is  dictated. 


III.  ACCEPTANCE  SAMPLING  PLAN 

The  armor  packages  tested  at  the  BRL  consist  of  a  right  side  and  a  left  side,  which  are 
designated  as  one  set.  One  month’s  production  is  considered  to  be  a  production  lot.  Every 
month  we  continue  testing  one  set  at  a  time  until  a  decision  can  be  made  about  that  produc¬ 
tion  lot.  For  a  given  set,  one  shot  is  fired  into  each  side;  and,  if  spacing  on  the  target  permits, 
a  second  shot  follows.  In  each  of  the  first  three  months,  a  total  of  at  least  four  shots  is 
required  in  order  to  make  a  decision  concerning  that  month’s  production.  This  provides  addi¬ 
tional  confidence  during  the  early  stages  of  the  plan.  There  are  two  portions  of  the  accep¬ 
tance  sampling  plan.  The  first  is  structural  integrity,  handled  using  attribute  methods;  the 
second  is  depth  of  penetration  of  a  particular  round  fired  into  the  armor,  handled  using  vari¬ 
able  techniques.  For  both  portions,  decisions  concerning  a  production  lot  should  be  based 
upon  the  data  from  all  available  shots  on  that  lot. 
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A  combined  chain  sampling  plan  was  proposed.  The  maximum  length  of  the  chain  was 
fixed  at  eight,  meaning  that  after  the  chain  has  been  established,  we  will  consider  the  current 
set  along  with  the  seven  immediately  preceding.  While  the  chain  is  growing,  there  is  an  area 
between  the  criteria  for  acceptance  and  rejection  in  which  we  can  make  no  decision.  At  least 
one  set  will  be  tested  each  month;  but  if  no  decision  can  be  made,  tightened  inspection  will 
dictate  the  examination  of  additional  sets,  possibly  up  to  a  maximum  of  eight.  Table  1  shows 
the  relationships  among  months,  sets,  and  shots  for  this  particular  procedure.  Note  that  the 
maximum  number  of  sets  and,  hence,  the  maximum  number  of  shots  decrease  over  time  as 
the  chain  is  being  formed.  Following  the  third  month  and  the  concurrent  drop  in  the 
minimum  number  of  shots,  then  when  the  chain  is  at  its  full  length  (definitely  by  the  eighth 
month),  one  set  and  at  most  four  shots  are  all  that  is  required  in  order  to  make  a  decision  for 
each  subsequent  production  lot. 

A  rejection  in  either  the  structural  integrity  or  the  penetration  depth  will  result  in  overall 
rejection  of  the  production  lot.  In  that  case  production  is  stopped,  adjustments  and  correc¬ 
tions  are  made,  and  testing  resumes  with  the  construction  of  a  new  chain.  If  neither  measure 
results  in  a  rejection  but  at  least  one  falls  within  the  no-decision  region,  another  set  should  be 
examined  and  both  categories  re-evaluated  using  the  additional  data. 

A.  Acceptance  Sampling  by  Attributes 

Projectiles  are  fired  at  these  packages,  which  are  then  inspected  for  structural  integrity. 
With  attribute  sampling,  only  two  outcomes  are  possible.  The  structural  integrity  is  assessed 
to  be  either  defective  or  non-defective,  regardless  of  the  number  of  shots.  Any  decision  to 
either  accept  or  reject  a  lot  is  based  on  the  number  of  defective  plates  in  the  sample  being 
considered. 

Chain  sampling  is  employed  in  this  attribute  sampling  plan.  Results  from  the  most 
recent  eight  sets  influence  decisions  regarding  a  lot.  A  lot  can  be  either  accepted  or  rejected 
at  any  time  (except  for  one  case  discussed  in  the  next  paragraph).  In  the  early  stages  of  sam¬ 
pling  there  is  also  an  area  in  between  acceptance  and  rejection  where  no  decision  is  rendered 
immediately  but  sampling  is  continued.  After  a  chain  reaches  its  full  length  of  eight  sets,  a 
decision  to  accept  or  reject  is  made  immediately. 

In  the  sampling  plan,  a  safeguard  is  built  in  to  prevent  rejection  of  a  good  lot  after  only 
one  set.  If  there  are  no  defectives  in  the  first  set,  the  lot  is  accepted.  Otherwise,  no  decision 
is  made.  Subsequently,  rejection  would  occur  only  when  there  were  three  or  more  failures  in 
the  most  recent  eight  sets. 

Table  2  shows  the  decision  rules  for  a  chain  building  to  a  maximum  length  of  eight.  The 
OC  curves  for  this  plan  are  depicted  in  Figure  1.  It  shows  that  for  a  chain  at  full  length,  the 
probability  of  accepting  a  lot  whose  true  defective  rate  is  5%  is  equal  to  0.96,  while  the  proba¬ 
bility  of  accepting  a  lot  whose  true  defective  rate  is  10%  is  equal  to  0.79.  Power  curves  for 
the  plan  are  depicted  in  Figurte  2.  For  a  chain  at  full  length,  the  probability  of  rejecting  a  lot 
from  a  process  whose  true  defective  rate  is  5%  is  equal  to  0.04,  while  the  probability  of  reject¬ 
ing  a  lot  whose  true  defective  rate  is  10%  is  equal  to  0.21.  (Note,  if  these  probabilities  are 
deemed  to  be  unsatisfactory,  a  different  plan  providing  more  satisfactory  levels  could  be 
developed  by  varying  the  maximum  chain  length  or  modifying  the  decision  rules). 
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TABLE  1.  Relationships  Among  Variables  in  Chain  Sampling  Procedure 


TABLE  2.  Decision  Rules  for  Acceptance  Sampling  by  Attributes. 


Operating  Characteristic  Curves 

Most  Recent  Eight  Sets 


Defective  Rate 


Power  Curves 

Most  Recent  Eight  Sets 


FIGURE  2.  Power  Curves  for  Acceptance  Sampling  by  Attributes. 


B.  Acceptance  Sampling  by  Variables 

When  primary  interest  is  in  a  process  level  rather  than  a  percent  defective,  sampling  by 
variables  is  the  proper  procedure.  For  the  armor  packages,  depth  of  penetration  for  a  partic¬ 
ular  munition  was  the  process  level  of  interest.  When  variable  sampling  plans  are  established, 
two  major  assumptions  must  be  satisfied:  first,  the  distribution  of  the  variable  of  interest 
must  be  known;  and  second,  a  good  estimate  of  its  standard  deviation  must  be  available. 

In  our  particular  problem  there  were  22  baseline  shots  from  which  we  were  to  determine 
a  distribution  and  estimate  its  standard  deviation,  as  well  as  establish  acceptable  and  reject- 
able  process  levels  (APL  Sc  RPL).  The  22  shots  had  a  mean  (X^)  of  5mm  with  a  standard 
deviation  (sb)  of  30mm.  The  data  had  been  transformed,  allowing  for  both  positive  and  nega¬ 
tive  penetration  values.  When  plotted,  the  data  appeared  normal;  and,  indeed,  the  hypothesis 
of  normality  could  not  be  rejected  using  statistical  goodness-of-fit  tests.  The  APL  was  esta¬ 
blished  at  20mm  (1/2  baseline  standard  deviation  from  the  baseline  mean)  and  the  RPL  was 
set  at  80mm  (2  1/2  baseline  standard  deviations  from  the  baseline  mean),  a,  the  probability 
of  rejecting  at  the  APL,  was  set  at  0.05;  and  /?,  the  probability  of  accepting  at  the  RPL,  was 
allowed  to  vary  with  the  sample  size  -  for  a  sample  of  size  four,  (3  would  equal  0.10. 

As  in  the  attribute  case,  a  set  consists  of  a  right  side  and  a  left  side.  For  each  set  an 
attempt  will  be  made  to  fire  a  second  round  into  each  side.  Because  this  might  not  always  be 
possible,  due  primarily  to  discrepancies  between  the  aim  point  and  the  hit  location,  each  set 
can  result  in  either  two,  three,  or  four  data  points,  depending  on  whether  or  not  both  shots  on 
each  side  are  considered  to  be  good  hits.  It  is  important  that  during  the  first  three  months, 
while  the  chain  is  being  formed,  at  least  four  shots  are  available  upon  which  to  make  a  deci¬ 
sion.  Table  3  outlines  the  decision  rules  for  the  variable  sampling  plan.  Like  the  attribute 
sampling  plan,  it  incorporates  chain  sampling  with  a  maximum  length  of  eight  sets.  The  plan 
will  not  reject  based  on  the  first  sample,  and  it  has  a  region  of  no  decision  until  the  chain 
reaches  its  full  length.  In  this  table,  X  represents  the  mean  penetration  depth  for  all  shots 
currently  considered,  s  represents  the  standard  deviation  of  this  sample,  n  is  the  total  number 
of  shots  used  in  computing  X,  and  1 9S  represents  the  95th  percentile  of  the  t-distribution  for 
the  appropriate  degrees  of  freedom  (n-1).  Thus,  n  can  vary  from  2  to  32  depending  upon  the 
length  of  the  chain  and  the  number  of  shots  available  on  each  side  of  the  armor  package. 

Because  n  varies  so  widely,  any  one  of  many  OC  curves  may  be  applicable.  Figure  3 
shows  these  curves  for  sample  sizes  2,  32,  and  many  integers  in  between.  TTie  abscissa  value, 
D,  represents  a  multiple  of  sb  from  Xb;  thus,  the  numbers  in  parentheses  are  the  penetration 
depths  in  millimeters.  Note  that  for  all  n,  the  probability  of  accepting  at  the  APL  is  0.95 
(1-a).  Because  the  probability  of  accepting  at  the  RPL  is  too  high  for  n»2  and  n»3,  the  pro¬ 
cedure  will  not  allow  lot  acceptance  at  these  small  sample  sizes  (see  Table  3).  Table  4  pro¬ 
vides  the  values  for  the  t-statistic  for  (l-a)-levels  of  0.99  and  0.95  and  degrees  of  freedom 
from  3  to  31. 

Power  curves  show  the  probability  of  rejecting  a  particular  lot.  Generally,  they  are  noth¬ 
ing  more  than  the  complement  of  OC  curves.  However,  for  our  procedure  this  is  not  the 
case,  since  there  is  a  region  of  no  decision.  Figure  4  shows  the  power  curves  for  this  variable 
sampling  procedure.  Basically,  there  are  two  sets  of  curves  -  the  first  two  pertaining  to  a  * 
0.05  and  the  next  three  pertaining  to  a  »  0.01.  Note  from  Table  3  that  in  order  to  reject 
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TABLE  3.  Decision  Rules  for  Acceptance  Sampling  by  Variables. 


DECISION  RULES 


SET 

NUMBER 

ACCEPT  * 

REJECT  * 

NO  DECISION 

1  (n  <  4) 

VMM 

ALL 

i  («  “  4) 

X-APL 

,r-  “'l9S 
s  /vn 

X-APL 

r  >t-95 
s /vn 

2  (combine  with  1) 

X-APL 

—  ^  95 

i /v^T 

X-APL 

lJ-  >*‘9S 
s /vn 

3  (combine  with  1,2) 

• 

X-APL 

~*'95 

s  /vn 

X-APL 

^  t  Art 

s  /Vn 

tT 

IV 

V 

VO 

Vi 

■ 

• 

7  (combine  with  1-6) 

X-APL 

s  /vn 

X-APL 

-  >  t  OA 

s  /Vn 

X-APL 

*.99^  r-  >  *.95 

s  /vn 

8  (combine  with  1-7) 

X-APL 

s /vn 

X-APL 

r  >  '.95 
s  /vn 

9  (combine  with  2*8) 

X-APL 

r  ^'.95 
s /vn 

X-APL 

r  >f« 
s /vn 

i  • 

• 

k  (combine  with  (k-7)  -  (k-1)) 

X-APL 

s  /vn 

X-APL 

^  t  95 

,  /'fit 

■  vw 

At  least  four  shots  are  required  in  each  of  the  first  three  months; 
otherwise,  regard  as  "No  Decision". 


Curves 


Curves  for  Acceptance  Sampling  by  Variables 


TABUE  4.  Values  of  the  Cumulative  t-Statistic 


•This  table  Is  abridged  from  Tables  of  the  Probability  Integral  of  the  Central  t-Disttibution 
by  R.E.  Mioduski,  BRL  Technical  Note  #1570,  August  1965. 
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Power  Curves 

One-Sided  t— Test 


Figure  4.  Power  Curves  for  Acceptance  Sampling  by  Variables. 


before  the  chain  is  at  its  maximum  length,  we  use  the  smaller  a-level,  and  Figure  4  shows 
some  possible  sample  sizes  for  a  *  0.01.  If  we  reject  at  an  cflevel  of  0.05,  our  sample  size 
must  be  somewhere  between  16  and  32;  and  these  curves  are  also  shown  in  Figure  4.  Gen* 
erally,  the  power  curves  are  of  more  interest  to  the  producer  than  the  OC  curves,  since  they 
highlight  the  producer's  risk. 

C.  Quality  Control  Charts 

Variations  in  the  manufacturing  process  are  either  random  or  assignable.  A  process  is 
"in  control"  when  only  random  variations  are  present.  Assignable  variations,  if  uncorrected, 
may  eventually  result  in  rejection  of  a  manufactured  lot.  However,  they  can  often  be 
identified  through  the  use  of  quality  control  charts. 

A  quality  control  chart  is  a  graphical  comparison  of  test  data  with  some  previously  com¬ 
puted  control  limits.  The  most  common  quality  control  chart  is  the  Shewhart.  chart,  named 
for  its  originator,  Dr.  Walter  A  Shewhart.  Figure  5  is  a  Shewhart  control  chart  for  mean 
penetration  depth,  the  variable  of  interest  in  our  armor  package  acceptance  sampling  plan. 
The  APL  is  the  central  line  with  an  upper  control  limit  equal  to  the  RPL,  two  baseline  stan¬ 
dard  deviations  away  from  the  APL  If  we  were  concerned  about  extremely  low  penetration 
depths,  we  would  incorporate  a  lower  control  limit  as  well.  Assuming  a  normal  distribution 
with  parameters  equul  to  those  of  the  baseline  data  implies  that  if  only  random  variations  are 
present,  99.38%  of  the  time  the  mean  penetration  depth  of  the  sample  will  fall  below  the 
upper  control  limit.  This  leaves  a  false-alarm  frequency  of  less  than  1%  (0.62%)  -  so  low  that 
this  control  limit  seems  to  be  a  reasonable  threshold  to  distinguish  between  random  varia¬ 
tions  and  assignable  variations. 

The  mean  penetration  depth  is  plotted  for  consecutive  sets  of  armor  plate.  If,  over  a 
period  of  time,  we  see  a  drifting  toward  the  control  limit,  the  process  can  be  examined  and 
adjusted.  This  might  possibly  eliminate  some  future  rejection  of  an  entire  lot. 

A  similar  chart  should  be  constructed  for  the  range  of  penetration  depth  within  the  sam¬ 
ple,  to  Insure  that  the  variability  of  the  armor  packages  is  not  increasing.  A  third  chart  for 
structural  integrity,  the  attribute  of  interest  in  our  acceptance  sampling  plan,  would  also  be 
useful.  In  each  cose  appropriate  upper  control  limits  must  be  established. 

Over  the  years  alternative  quality  control  charts  have  emerged,  each  with  their  own  set 
of  advantages  and  disadvantages.  One  of  the  most  popular  has  been  the  cumulative  sum  con¬ 
trol  chart  (cusum  chart).  Here,  decisions  are  made  based  on  all  the  data  rather  than  just  the 
last  sample.  An  advantage  of  the  cusum  chart  is  that  it  often  displays  sudden  and  persistent 
changes  in  the  process  mean  more  readily  (that  is,  with  fewer  samples  and  less  expense)  than 
a  comparable  Shewhart  chart.  However,  control  limits  are  somewhat  less  intuitive  and, 
therefore,  more  difficult  to  establish.  Somewhere  in  between  the  Shewhart  chart  and  the 
cusum  chart  are  quality  control  charts  that  use  some,  but  not  all,  of  the  past  data.  Many  of 
these  techniques  incorporate  a  weighting  factor,  providing  more  weight  to  the  most  recent 
data. 
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Shewhart  Control  Chart 


FIGURE  5.  Quality  Control  Chart  for  Mean  Penetration 


It  is  important  that  some  type  of  quality  control  charts  be  represented  in  the  acceptance 
sampling  plan.  They  are  relatively  easy  to  maintain  and  might  provide  early  warning  signs 
which  could  be  beneficial  to  both  the  producer  and  the  consumer. 


IV.  CONCLUSIONS 

Generally,  it  is  not  feasible  for  a  consumer  to  inspect  every  item  from  a  production  lot 
that  he  might  want  to  purchase.  A  judicious  choice  of  a  lot  acceptance  plan  will  allow  him  to 
sample  the  production  lot  and  determine  with  a  pre-established  level  of  confidence  whether 
or  not  it  meets  his  specifications,  Chain  sampling  is  a  particular  method  of  lot  acceptance 
sampling  used  when  sample  sizes  are  small.  It  utilizes  the  most  recent  lot  information  to  pro¬ 
vide  more  confidence  in  the  decision. 

In  testing  armor  packages  for  acceptance  by  the  US  Army,  chain  sampling  provides  a 
logical  method,  since  destructive  testing  dictates  small  sample  sizes.  A  technique  involving 
both  structural  integrity  (attribute  sampling)  and  penetration  depth  (variable  sampling)  has 
been  proposed.  One  set  of  armor  packages  is  represented  by  both  a  left  side  and  a  right  side. 
The  procedure  allows  for  accepting  the  production  lot  (one  month’s  production)  after  exa¬ 
mining  just  one  set.  It  allows  for  rejecting  the  production  lot  only  after  testing  at  least  two 
sets.  There  is.  a  region  of  no  decision;  but  after  the  chain  has  reached  its  maximum  length  of 
eight  sets,  a  decision  must  be  rendered. 

Operating  characteristic  curves  and  power  curves  provide  the  probability  of  accepting 
and  rejecting  lots  given  a  percent  structurally  defective  (attributes)  and  given  a  mean  penetra¬ 
tion  depth  (variables). 

In  addition  to  the  acceptance  sampling  plaas,  control  charts  should  be  used  for  both  the 
attribute  and  variable  parameters.  These  charts  display  sample  results  for  particular  parame¬ 
ters  such  as  percent  defective,  mean  penetration  depth,  and  variability  of  penetration  depth. 
The  data  might  be  presented  as  individual  sample  points  or  as  sums  over  a  preceding  number 
of  samples.  By  continually  examining  the  control  charts,  we  can  see  when  one  of  the  parame¬ 
ters  is  drifting  toward  the  rejection  region,  enabling  the  producer  to  make  adjustments  and, 
possibly,  preventing  rejection  of  an  entire  lot  of  armor  plate. 

The  proposed  lot  acceptance  plan  was  briefed  to  the  Project  Manager  M1A1  on  14  April 
1988  at  Aberdeen  Proving  Ground,  Maryland.  It  was  approved  and  will  be  adopted  subject  to 
any  refinements  agreed  upon  by  both  the  US  Army  Ballistic  Research  Laboratory  and  the 
Project  Manager. 
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SOME  NOTES  ON  VARIABLE  SELECTION 
CRITERIA  FOR  REGRESSION  MODELS 
(AN  OVERVIEW) 


Eugene  F .  Du  to  I t 
U.S.  Army  Infantry  School 
Fort  Banning,  Georgia 


Abstract .  There  are  several  decision  rules  for  determining  whan 
to  enter  additional  independent  variables  into  linear  multiple 
regression.  Three  of  these  are:  (1)  examining  the  incremental 
significance  in  the  multiple  correlation  cost f Icients,  (2) 
Mallows'  CM  statistic  to  determine  the  beat  combination  of 
Independent  variables,  and  (3)  considering  the  changes  In 
magnitude  or  the  standard  arror  of  estimate.  This  paper  will 
examine  some  of  the  i  nter re  I  at  I onah 1 pe  between  the  three  methods 
oitod  above.  These  relationships  will  ba  applied  to  a  data  set 
and  the  results  presented. 

Acknowl adaement .  The  author  wishes  to  thank  Professors  Ron 
Hocking  and  Emanuel  Parzen  for  their  comments  and  suggestions. 

It  is  this  spirit  of  freely  shared  Ideas  that  makes  these 
"Design  of  Experiments  Conferences"  valuable  to  Army 
stat  1st Ic I ans. 

1 .  The  Problem. 


Given  experimental  data  In  the  form: 


Y 

Xi 

Xa 

.  _  x* . 

•  •  • 

X. 

Y  i 

Xi  , 

Xi  a 

X  i  a 

•  •  t 

Xi  a 

Ya 

X  a  i 

Xaa 

Xaa 

•  •  « 

Xa* 

Y» 

X  a i 

Xaa 

Xaa 

•  •  « 

Xa« 

Y  n 

Xml 

Xria 

Xna 

...  X  n  m 

where 

(Xi  , 

Xa, 

X»,  . 

.  • ,  X,) 

are  candidate 

1 ndependant 

var 

1  ab 1 es 

(that  make 

sense  according  to 

some 

theor at 

i  ca  1 

bases)  and 

Y  is  the 

dependant  variable. 

The 

researcher  wants  to  form  some  model 
k 

V '  ■  a  +  bi  X,  ( 1  ) 

1-1 

where  k<  q.  This  would  indicate  the  model  (aquation  1) 
consisted  of  the  best  set  of  candidate  independent  variables. 
This  paper  wl  I  I  provide  an  overview  of  the  fol  lowing  measures 
and  criteria  In  order  to  shed  some  light  on  this  problem. 
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2.  Tha  Multiple  Correlation  Coefficient.  (Ra) 


a.  Incremental  S I gn 1 f i canoa,  The  wall  known  taat  for  tha 
inoremantal  algnlfloanca  In  Ra  by  adding  an  additional 
Indapandant  var labia  X*  Into  aquation  (1)  la: 

F  •  (R^.i.a.,  ..tei  -  .  a  .  .  .  ka.)__7-  _(_Kj  ka)  (2) 

(1  -  R*v . i , a, . . . ki )  /  (N  -  Ki  -  1) 

whara  ki  *  numbar  of  Indapandant  variables  for  largar  Ra 
ka  ■  numbar  of  Indapandant  variablaa  tor  amallar  Ra 
Rav . i , a .... m  *  I argar  Ra 
Rav. i , a. . . .ka  *  amallar  Ra 
M  ■  numbar  of  caaaa 

Tha  taat  follows  an  F  distribution  with  dsgrsss  of  fraadom  aqua  I 
to  (ki  -  ka)  ,  (N  -  ki  -  1 )  . 

b.  Ad  Jus tad  Ra ,  As  Indapandant  variablaa  are  added  to 
aquation  (1),  tha  value  of  Ra  will  alao  increase.  This 
Inoraasa  may  be  small  (l.a.,  atat 1st  lost  I y  not  significant).  In 
ordsr  to  aooount  for  this  mathematical  inoraasa  In  Ra,  tha 
so-called  shrinkage  formula  Is  used  to  calculate  an  adjusted 

Ra  as: 

Ra  adj  «  Rak  -  k  (1  -  Rak )  (3) 

N  -  k  -  1 

where  k  ■  numbar  of  Independent  variables  In  regression. 

N  ■  numbar  of  oasaa. 

3.  Mai  Iowa*  Co  Statistic.  Myara  (reference  1)  presents  tha  Cp 
statistic  In  tha  following  form: 

Cp  ■  p  +  (Sa  -  **)  (  N  -  p)  (4) 

<ra 

where  p  ■  k  +  1 

jj?  ■  estimated  variance  of  tha  oomplete  model  (l.a.; 
u  ail  Indapandant  variables  included). 

Sa  ■  oat  Imiitad  variance  of  tha  candidate  (subset) 
mode  1  . 

N  a  numbar  of  oases. 

and  Sa  are  obtained  as  tha  residual  mean  squares  from 
tha  regress  I  on  ANOVA . 

Tha  following  Interpretation  Is  baaed  on  tha  discussion  from  the 
Myara  (reference  O  text: 


220 


F i gur e  1 


1  2  3  4  r  P 


Reference  to  aquation  (4)  shows  that  If  Sa  <  (f  a ,  the  plot 
of  Cp  w  i  II  fall  below  tha  II na  Cp  ■  p.  Tha  above  I naqua I  I ty  la 
daelrable  for  It  atataa  that  tha  variation  about  aome  subset 
regression  modal  Is  lass  than  tha  variation  about  the  full 
model.  Only  point  C  In  tha  abova  diagram  meats  this  condition. 
This  concept  will  ba  dlsouaaad  in  the  following  paragraph 
concerning  tha  standard  error  of  estimate  about  regression.  It 
should  be  noted  that  If  {Sa  *  tfa) ,  then  equation  (4)  becomes 
Cp  ■  p.  This  Is  always  tha  case  for  tha  full  modal.  An 
alternative  format  for  Cp  Is  given  by  Daniel  and  Wood  (rafaranoa 
2)  as: 

cp  ■  RSSt,  -  (N  -  2  p]  (5) 

£Ta 

where  p  ■  k  +  )  (as  before) 

RSSk,.  »  residual  sum  of  squares  with  k  Independent  variable 
(p  parameters) 

0*  a  ■  residual  mean  square  of  the  complete  model  (as 
before)  . 

It  can  be  shown  (not  here)  that  equation  (6)  la  equivalent  to 
equal  Ion  (4)  . 

3A.  Another  Alternative  Form  for  Cp.  Given  q  independent 
variables,  the  total  regression  model  is: 

Y  “  a  +  bi Xi  +  baXa+. • .+b*X*  (6) 

The  regression  ANOVA  table  Is  presented  below  as  Figure  2. 

Figure  2 

Regression  ANOVA  -  Full  Model 
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The  mode)  with  k  independent  variables: 


where  k  <  q  is: 


^  *  a  +  bi  Xi  +  ba  Xa  +  .  .  .  bk.:  Xu  ( 7 ) 

The  regression  ANOVA  table  Is  given  In  Figure  3. 

F i gur e  3 

Regression  ANOVA  -  Subset  Mode! 


Source 
Exp  I e i ned 
Res  I  due  I 


DF  SS  MS 

k  (N-1)  (Say)  (R*k, ) 

N-k-1  (N-k-1)  S*y|XK  S a y  | « ki' 

N  -  1 


where 

Say  |  Xk  -  (Say)  ( I  -Ra>„ ) 

N-k-1 

Say|X„  -  fjHJ.  (Say)  ( 1  -R*„) 

N-q-  1 

Referring  to  the  Myers  format  of  Cp  (equation  4)  end 
aubatltutlng  equations  (6)  for  S*  end  a: 

Cp  ■  (k+1)  +  (Sav  i Xh,  -  SaYIX-)  (N-k-1) 

SayH 


(8) 


Further  substitution  [referenoe  equations  (8)]  end  simple 
algebra  yield  another  format  for  the  Cp  test: 


CP  *  (N-q-  1  )  (  1  -Ra>,; )  -  (N-2k  -  2)  (8) 

( 1 -Ra„) 


Note  that  In  various  forms  the  Cp  test  can  be  expressed  as  a 
function  of  Ft®,  Say  X,  N,  q,  k.  Thl#  leads  to  another 
Independent  decision  method,  namely  the  standard  error  of 
regress  i  on  (Syl  X)  . 

4.  5  y  I X  >,  ,  When  performing  step-wise  regression,  the  value  of 

Sa Y I X  usually  gats  smaller  as  independent  variables  nru  added 
to  regression.  In  other  words 

SaY  I  Xk,  >  SaY  |  X**i  (usually).  (10) 


However,  this  Is  not  always  the  case.  The  ratio  Is  actually 
SaY I Xto  >  . 

SaY I Xu»i  <  1  (11) 


Note  that  equation  M)  a  I  I  owe  for  cases  where  some  subset  model 
has  I eaa  variance  (Sa)  than  the  variance  for  the  oompleta 
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model  (g-a).  In  this  case  the  Cp  plot  is  below  the  line 
(i.e.;  point  C  in  Figure  1).  This  can  be  expressed  in  Figure  4: 


F I gur a  4 


The  minimum  value  In  Figure  4  corresponds  to  point  C  In  Figure 
1.  The  subnet  of  regression  that  hss  minimum  variance  would  be 
the  best  predictor  of  the  dependent  variable  V.  The  ratio 
deaorlbed  by  equation  (11)  can  be  rewritten  to  gain  some  Insight 
to  the  process.  From  equations  (8)  tha  following  expressions 
can  be  Inferred: 


SaY|X>,  “  N-1  (S»v>  ( 1  -Ra».) 

N-k-1 

S*YIX^i  ■«  N-1  <SaY)  ( 1  -Rak,  *i  ) 

N-k-2 


(  12) 


The  ratio  In  Inequality  (11)  now  becomes: 

SaY I  X,,  ■  (N-k-2)  .  1  -IVk  (13) 

SaY | Xi * i  (N-k-1)  1 -R*k * i 

where  N-k-2  <  1 

N-k-  1 

and  1  -  R a  >■  >  1 

1  -Rak,'*i 

Therefore,  the  value  of  the  ratio  clteo  as  inequality  (11) 
will  depend  on  the  magnitudes  of  the  ratios  shown  above.  Note 
that  equation  (3)  (R-.«j)  contains  a  shrinkage  factor  with 

terms  (l‘*Ra»)  and  (N-k-1).  These  terms  are  else  contained 
in  equation  (13).  Intuitively  it  appears  that  the  adjusted 
correlation  coefficient  (R*»«j)  should  be  a  maximum  for  the 
subset  of  regression  where  SaY|X  le  a  minimum.  The  value  of 
Cp  should  also  be  minimum  for  the  same  subset  of  Independent 
var 1 ab I es . 

5.  Examp  I e ■  The  fol  lowing  example  wao  taken  from  Myers 
(reference  1).  It  Is  found  on  page  110,  Table  4-1.  The  example 
uses  sales  data  for  asphalt  shingles  obtained  from  (N®15) 
districts.  The  variables  considered  in  this  example  are: 
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Xi  =  Number  of  promotional  accounts. 

Xa  =  Number  of  active  accounts. 

X*  *  Number  of  competing  brands. 

X «  *  District  potential. 

Y  ■  Sales  in  thousands  of  dollars. 

The  results  of  s  step  wise  regression  are  given  below: 

Figure  5 

Stepwise  Results 


|  STEP 

VARIABLE 

SY  |  X„  #  1 

ADJ  Ra  *  j 

P  I 

» _ 1 

(3 

■WM 

|U| 

2.3  . 

HkH 

■RE**  1 m 

3 

1.2.3 

4.98 

.99612 

4 

mrwm 

4 

1  .  2  .  3  .  4 

6 . 12 

.99590 

5 

5 

Notice  that  the  last  step  (number  4)  has  s  Cp  value  equal  to  p 
(5).  This  is  always  true  for  the  full  model.  Also  note  that 
step  3  Is  the  best  subset  regression.  It  Is  this  step 
(variables  Xi ,  Xa,  X*,)  where  the  values  of  Sy| x  and  op 
are  minimum  and  adjusted  Ra  is  maximum. 

The  results  of  all  variable  oases  la  presented  In  Figure  6 
The  combination  shown  In  step  It  Is  the  best  subset  regression 
It  is  the  same  combination  of  optimum  values  of  RaadJ ,  Sy|x 
and  Cp  (variables  Xi  ,  Xa ,  Xs,). 

Figure  6 

A I  I  Cases 


STEP 

VARIABLE 

HB« 

Ra 

4 

ADJ  R® 

19 

mm 

1 

82.49 

.01200 

- . 06400 

2 

3361 

2 

58.63 

.50101 

.46263 

2 

1692 

3 

49.99 

.63725 

.60935 

2 

1227.1 

4 

79.05 

. 09284 

.  02306 

2 

3065 . 1 

5 

1  .2 

60.53 

.50900 

.42716 

3 

1666.8 

6 

1  ,3 

51.71 

.64169 

.58197 

3 

1213.9 

7 

1  .4 

81.26 

.11609 

-.03240 

3 

3011.2 

8 

2,3 

6.67 

.99403 

.99303 

3 

11.4 

9 

2,4 

60.82 

.50422 

.42159 

3 

1683.1 

_LG _ 

_ 3^4 _ 

48.83 

■f-I-M-ui 

3 

IM-1K1 

6.  Summary .  This  paper  has  examined  several  methods  of 
determining  when  to  enter  additional  Indapendant  variables  into 
I  Inear  multiple  regression  In  order  to  form  a  optimum  subset 
from  all  the  candidate  variables. 

The  I nter re  I  at  I onsh I pe  between  Cp,  Syj x  and  adjusted  R* 
are  studied.  These  three  Indicators  appear  to  provide  the  same 
information  In  the  model  selection  decision  process.  Although 
they  all  lead  to  the  same  decision  regarding  the  subset 
regression  selection,  each  measure  provides  a  dlffarant 
perception  on  the  subject. 
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TWO-STAGE  TESTING  OF  COMBAT  VEHICLE  TIRE  SYSTEMS 


Barry  A.  Bodt 


US  Army  Ballistic  Research  Laboratory 
Aberdeen  Proving  Ground,  Maryland  21005-5066 


ABSTRACT 

An  effort  is  underway  to  enhance  the  battlefield  survivability  of  combat  vehicle  tire  sys¬ 
tems.  The  Army  is  currently  investigating  several  new  tire  technologies  with  regard  to  their 
ability  to  function  after  battlefield  degradation.  The  tires,  in  a  run-flat  condition,  must  sup¬ 
port  some  vehicle  mobility  according  to  that  vehicle’s  mission  profile.  The  immediate  objec¬ 
tive  of  this  program  is  choosing,  for  further  research,  the  most  promising  among  the  new  tire 
technologies.  The  presenter  has  been  tasked  to  develop  an  appropriate  test  plan, 

Sound  experimental  strategy,  for  this  or  any  study,  must  be  accompanied  by  a  clear 
understanding  of  the  problem(s)  to  be  resolved.  A  list  of  question  areas  worth  exploring  to 
help  gain  this  understanding  is  suggested  by  Hahn  (Technometrics,  1984)  as  part  of  more 
general  guidelines.  The  presenter  demonstrates  their  usefulness  to  that  end  in  the  above 
mentioned  tire  program.  The  test  plan  and  the  process  by  which  it  evolved  is  discussed. 
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TWO-STAGE  TESTING  OF  COMBAT  VEHICLE  TIRE  SYSTEMS 
1.  INTRODUCTION 

An  effort  is  underway  to  enhance  the  battlefield  survivability  of  combat  vehicle  tire  sys¬ 
tems.  The  impetus  for  current  investigations  dates  back  to  a  1979  issue  paper,  submitted  to 
DA  by  the  US  Training  and  Doctrine  Command  (TRADOC).  In  1985  the  Tank  Automotive 
Command  (TACOM)  established  a  tire  task  force,  the  need  for  which  was  supported  by  the 
results  of  a  1984  independent  evaluation  of  one  tire  system  performed  by  the  Operational 
Test  and  Evaluation  Agency  (OTEA).  OTEA  observed  that  when  the  run-flat  tires  for  the 
High  Mobility  Multi-Purpose  Wheeled  Vehicle  (HMMWV)  were  run  flat  for  30  miles,  the 
tires  became  unserviceable  and  had  to  be  replaced.  The  objective  of  the  TACOM  Tire  Task 
Force  is  to  identify  a  survivable  tire  system  (STS)  technological  replacement  which  demon¬ 
strates  acceptable  battlefield  survivability.  A  two-phase  approach  (operational  and  technical) 
has  been  adopted  to  screen  available  STS  technologies  in  search  of  candidates  for  more 
intense  research  and  development.  The  operational  phase,  considering  the  standard  and 
seven  new  STS  technologies,  was  completed  by  the  Combat  Developments  Experimentation 
Center  (CDEC)  in  1987.  The  technical  phase,  the  focus  of  this  paper,  is  being  conducted  by 
the  Vulnerability  Lethality  Division  (VLD)  of  the  Ballistic  Research  Laboratory  (BRL) 
according  to  the  test  plan  developed  by  the  Probability  and  Statistics  Branch  (PSB)  of  the 
Systems  Engineering  and  Concepts  Analysis  Division  (SECAD)  of  the  BRL 

This  paper  is  intended  to  accomplish  two  tasks.  The  first  is  to  discuss  the  test  plan  that 
has  been  adopted  for  the  technical  testing  phase  -  not  in  great  detail  but  sufficiently  to 
demonstrate  the  degree  to  which  experimental  objectives  are  satisfied.  As  part  of  the  discus- 
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sion  it  is  shown  how,  for  example,  tire  performance  specifications,  factors  thought  to 
influence  performance,  and  physical  and  budgetary  constraints  are  incorporated  in  the  test 
strategy.  The  second  is  to  illustrate  the  usefulness  of  well-defined  consulting  guidelines  in 
extracting  the  necessary  information  from  experimenters.  Any  sound  experimental  strategy 
must  be  accompanied  by  a  clear  understanding  of  the  problem  to  be  resolved,  but  informa¬ 
tion  essential  to  that  understanding  is  often  difficult  to  obtain.  The  fragmented  manner  in 
which  information  is  passed  from  client  to  consultant  inhibits  a  cogent  assimilation  of  facts 
needed  for  efficient  problem  solving.  Hahn  (1984)  suggests  imposing  the  structure  of  ques¬ 
tion  area  guidelines  (see  Figure  1)  both  to  help  sort  the  information  coming  in  and  to  direct 
consultation  sessions  down  new  promising  paths. 

The  remainder  of  the  paper  is  organized  as  follows.  In  Section  2  the  problem  and  test 
plan  are  developed,  punctuated  by  Hahn’s  guidelines.  It  is  hoped  that  this  presentation  will 
both  give  fair  treatment  to  the  Army  concern  as  well  as  illustrate  a  reasonable  approach  to 
consultation.  In  Section  3  a  brief  critique  of  the  test  plan’s  strengths  and  weaknesses  is  given, 
followed  by  some  closing  comments. 

2.  EVALUATION  OF  THE  TEST  PLAN 

Problem  information  is  divulged  in  this  section  according  to  Hahn’s  guidelines,  and  that 
constitutes  our  presentation  of  his  technique.  We  seek  only  to  show  how  encompassing  those 
question  areas  are  by  developing  in  full  the  Army’s  problem  through  their  use.  In  the  text, 
italicized  words  and  phrases  refer  buck  to  guidelines  in  Figure  1.  The  guidelines  have  been 
juggled  to  allow  for  a  logical  presentation  and  the  order  shown  in  Figure  1  corresponds,  with 
few  exceptions,  to  that  of  this  section.  This  is  simply  a  matter  of  convenience  and  not  a  claim 
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1.  The  objectives  of  the  experiment. 

2.  The  variables  to  be  held  constant  and  how  this  will 
be  accomplished  (as  well  as  those  that  are  to  be 
varied). 

3.  The  uncontrolled  variables  -  what  they  are  and 
which  ones  are  measurable. 

4.  The  response  variables  and  how  they  will  be  meas¬ 
ured. 

5.  Special  considerations  which  indirectly  impose 
experimental  constraints. 

6.  The  budged  size  of  the  experiment  and  the  dead¬ 
lines  that  must  be  met. 

7.  Conditions  within  the  experimental  region  where  the 
expected  outcome  is  known;  the  anticipated  perfor¬ 
mance  is  expected  to  be  inferior,  especially  for  pro¬ 
grams  where  an  optimum  is  sought;  and  experimen¬ 
tation  is  impossible  or  unsafe. 

8.  Past  test  data  and,  especially,  any  information  about 
different  types  of  repeatability. 

9.  The  desirability  and  opportunities  for  running  the 
experiment  in  stages. 

10.  The  anticipated  complexity  of  the  relationship 
between  the  experimental  variables  and  the 
response  variables  and  any  anticipated  interactions. 

11.  The  procedures  for  running  a  test,  including  the  ease 
with  which  each  of  the  variables  can  be  changed 
from  one  run  to  the  next. 

12.  The  details  of  the  physical  set-up. 


Figure  1.  Important  Question  Areas  for  Statisticians  to  Address. 
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for  an  ideal  sequence.  In  fact,  each  consulting  session  is  likely  not  only  to  naturally  gravitate 
toward  different  orders  but  also  to  move  around  from  area  to  area,  possibly  returning  several 
times  to  some. 

2.1.  Understanding  the  Froblem 

Let  us  begin  by  considering  objectives.  We  consider  two  types;  military  and  experimen¬ 
tal.  The  military  objective  is  that  HMMWV  tires  remain  serviceable  when  degraded  through 
battlefield  exposure  to  small  caliber  munitions  and  shell  fragments.  Serviceable  means  that 
the  tire  exhibits  performance  consistent  with  the  standards  specified  in  the  NATO-Finabel  20 
A  5  1956  NATO  Test  Procedure.  Summarized  expectations  set  forth  therein  say  that  the 
combat  tire  must  possess  (as  nearly  as  possible)  the  same  over  the  road  performance  as  the 
classic  radial  tire  in  terms  of  maximum  vehicle  speed  and  lateral  and  longitudinal  traction  and 
stability.  After  degradation,  normal  military  performance  of  the  vehicle  is  still  required  when 
no  more  than  two  tires  (one  drive  and  one  steering)  are  damaged.  The  experimental  objective 
is  to  screen  six,  including  the  standard,  tire  systems  with  the  purpose  of  selecting  a  subset  for 
further  research,  development,  and  the  eventual  upgrading  of  combat  tires.  The  selection  cri¬ 
teria  must  be  driven  by  the  military  objectives  summarized  above. 

Question  areas  2-4  in  Figure  1  each  concern  variables.  It  is  in  the  identification  and 
classification  of  these  variables  that  the  experimental  strategy  begins  to  take  form.  In  Table  1 
the  most  important  ones  are  given.  Care  is  taken  to  initially  classify  them  as  candidates  for 
response,  design  or  nuisance  variables  and  to  subclassify  them  for  each  of  the  last  two 
categories.  The  scale  of  measurement  is  also  noted.  A  short  definition  of  each  of  these  vari¬ 
ables  is  given  in  the  appendix.  Because  the  variables  listed  in  Table  1  represent  only  those 
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Table  1.  Relevant  Variables. 


fragment  velocity 


considered  essential,  all  must  be  incorporated  in  the  experimental  strategy.  We  briefly  dis¬ 
cuss  several  cf  them  here  so  that  the  reader  may  gain  a  sense  of  the  complexity  of  the  prob¬ 
lem. 


The  logical  starting  point  for  discussion  is  with  tire  technology,  for  it  is  the  selection 
from  among  these  prototypes  that  is  the  objective  of  this  experiment.  Six  manufacturer 
offerings,  including  the  standard,  are  to  be  considered,  but  there  are  basically  only  four  tech¬ 
nologies.  When  combat  tires  are  exposed  to  small  caliber  munitions  and  shell  fragments  they 
will  surely  tear,  puncture,  or  in  some  other  way  be  damaged  so  as  to  induce  partial  or  com¬ 
plete  deflation.  Then  in  order  for  military  objectives  to  be  satisfied,  the  survivable  tire  will 
either  successfully  negate  this  damage  or  be  structurally  capable  of  supporting  vehicle  mobil¬ 
ity  without  benefit  of  full  tire  pressure.  Taking  the  first  tact,  the  sealant  tire  systems  contain 
chemical  compounds  which  are  intended  to  flow  to  a  source  of  air  loss,  solidify,  and  thereby 
negate  the  threat  damage.  Run-flats  take  the  second  tact  and  are  able  to  support  the  vehicle 
with  a  metal  or  plastic  insert  which  acts  in  the  tires  stead  when  the  tire  is  deflated.  Self- 
supporting  tires  are  so  named  because  molded  into  the  tread  is  a  rigid  fiber  glass  band, 
designed  to  carry  the  tire’s  full  load  in  the  absence  of  tire  pressure.  Solid  urethane  tires  cir¬ 
cumvent  the  problem  by  containing  no  air  to  be  lost,  but  they  do  so  at  the  cost  of  additional 
weight,  inhibiting  vehicle  mobility  (Drelling  et  al.,  1987). 

A  limitation  of  the  CDEC  exercise  is  that  tire  degradation  from  shell  fragments  is  not 
considered.  Interest  in  the  more  irregular  pi '  i  :s  and  tears  caused  by  the  shell  fragment 
threat  is  a  consideration  in  involving  the  BRL  in  the  technical  phase  of  experimentation.  To 
make  inferences  about  tire  performance  after  fragment  damage  the  consensus  is  that  either 
live  shells  should  be  detonated  near  standing  tires  or  the  expected  resulting  fragments  should 
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be  simulated.  A  special  consideration  in  long  range  plans  is  that  an  acceptance  test  consistent 
with  current  testing  be  developed.  Due  to  the  repeatability  requirements  inherent  in  accep¬ 
tance  testing  the  shell-detonation  approach  was  dropped  in  favor  of  fragment  simulation 

This  decision  led  to  variables  involving  fragment  shape,  size,  and  velocity.  Due  to  budget 
and  time  constraints  it  appears  unreasonable  to  select  many  values  for  each  and  then  proceed 
in  a  factorial  manner  when  incorporating  them  in  the  design.  Rather  we  option  to  create  two 
fragment  profiles,  each  representative  of  a  distinct  threat.  Avoiding  great  detail,  a  standard 
BRL  fragment  design  is  specified  for  shape.  Velocity  and  mass  are  determined  as  follows. 
Each  are  a  function  of  the  distance  between  the  shell  at  detonation  and  the  tire.  The  distance 
selected  corresponds  to  a  50%  mobility  kill  for  the  vehicle  according  to  models  accessed  by 
the  Army  Material  Systems  Analysis  Activity  (AMSAA).  Avoiding  the  experimental  region 
where  the  expected  outcome  is  known,  we  do  not  consider  distances  so  close  that  the  personnel 
are  not  likely  to  survive.  The  median  velocity  and  mass  among  computer  simulated  frag¬ 
ments  possessing  an  appropriate  trajectory  then  serve  as  representative  values  for  these 
characteristics.  Trial  firings  suggest  some  deviations  in  these  choices  so  that  resulting  dam¬ 
age  seemed  similar  to  actual  fragment  damage  previously  obseived. 

Other  factors  of  keen  interest  include  the  terrain  traveled  and  tire  position,  motion,  and 
pressure.  The  mission  profile  for  the  HMMWV  dictates  that  it  shall  be  able  to  travel  primary 
roads,  secondary  roads,  and  cross  country.  Further  it  suggests  that  in  a  characteristic  mission 
those  three  terrains  might  comprise  30%,  30%,  and  40%,  respectively,  of  the  total  mileage 
covered  Tire  position  refers  to  its  designation  as  a  drive  tire  or  a  drive  and  steering  tire;  the 
HMMWV  is  4-wheel  drive.  In  addition  to  this  one-at-a-time  damage,  recall  that  the  NATO 
Finabel  standards  require  acceptable  performance  when  two  tires  on  the  vehicle  are 
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damaged.  When  attacked,  the  HMMWV  may  be  moving  or  at  rest.  Proponents  of  the 
sealant  technology  claim  that  if  the  tire  is  in  motion  when  punctured,  then  the  sealant 
mechanism  will  be  more  effective  in  finding  and  repairing  the  damage.  Past  test  data  indi¬ 
cates  that  tire  pressure  may  influence  the  type  of  puncture,  that  is,  clean  or  ragged.  Manufac¬ 
turer  recommended  high  and  low  pressures  for  each  tire  will  be  considered. 

The  special  consideration  that  this  experiment  complement  the  CDEC  exercise  fixed  two 
important  test  definitions.  TACOM  decided  that  the  response  would  remain  defined  as  miles 
until  failure.  Failure  occurs  when  either  the  tire  begins  to  come  apart  when  in  use  or  the 
operator  must  slow  to  less  than  50%  of  normal  operating  speed  in  order  to  maintain  control. 
Under  a  rigid  value  for  normal  speed,  failure  could  depend  on  the  size  and  strength  of  the 
operator.  We  propose  to  account  for  that  by  establishing  a  profile  on  operators  (actually  driv¬ 
ing  teams)  in  their  normal  operation  of  the  vehicle.  The  50%  rule  is  then  based  on  normal 
team  performance.  Driving  teams  are  established  to  avoid  failure  due  to  fatigue.  Past  test 
data  reveals  that,  some  degraded  tires  remain  serviceable  after  100  continuous  miles  of  opera¬ 
tion.  In  order  to  avoid  truncated  data,  the  test  course  is  extended  to  125  continuous  miles,  but 
at  the  additional  cost  of  trial  time.  It  is  felt  that  if  two  operators  are  allowed  to  rotate  after 
each  25  mile  lap,  then  fatigue  will  not  enter  into  the  failure  determination. 

2.2  Test  Plan 

The  test  plan  will  be  implemented  in  stages.  A  fairly  large  number  of  experimental  con¬ 
ditions  define  the  experiment  outlined  in  Section  2.1.  To  examine  each  of  these  conditions  in 
a  factorial  manner  will  require  mote  resources  than  the  experimental  budget  will  allow;  for  all 
but  the  standard  tire  no  more  than  30  prototypes  will  be  made  available.  Moreover,  recall 
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that  the  principal  objective  of  this  study  is  to  facilitate  comparison  among  tires.  Placing  too 
much  emphasis  (sample  allocation)  on  ancillary  issues  may  partially  obscure  (weaken  conclu¬ 
sions  regarding)  the  main  experimental  focus.  For  these  reasons,  resource  limitations  and 
emphasis,  we  choose  to  run  the  experiment  in  two  primary  stages. 

The  division  of  testing  meets  the  above  concerns.  In  stage  1  all  the  experimental  condi¬ 
tions  are  incorporated  in  the  design  as  factors  or  fixed  test  procedures.  Only  the  standard 
HMMWV  tire  is  considered  in  stage  1.  The  purpose  of  this  stage  is  two-fold.  First  the  vari¬ 
ous  test  conditions  may  be  examined.  It  is  hoped  that  some  will  prove  unnecessary  for  inclu¬ 
sion  in  stage  2,  thereby  increasing  the  experimental  information  per  sampled  test  condition. 
Second,  test  procedures  may  be  smoothed.  Field  test  exercises  nearly  always  present  unex¬ 
pected  problems,  often  resulting  in  samples  which  must  be  invalidated  for  the  analysis.  Here 
we  run  only  the  risk  of  wasting  some  more  plentiful,  standard  tires  instead  of  the  scarce  new 
prototypes.  In  stage  2  the  prototypes  will  be  examined  by  an  experienced  testing  group  under 
the  conditions  remaining  after  stage  1.  Since  the  complete  details  will  not  be  available  until 
stage  1  is  concluded,  we  defer  further  discussion  of  stage  2  to  future  papers.  In  the  remainder 
of  this  paper  stage  1  testing  serves  as  the  main  focus. 

Stage  1  will  be  run  as  a  1/2  replication  of  a  4x24  factorial  design,  requiring  32  observa¬ 
tions.  The  design  factors,  each  discussed  in  Section  2.1,  are  listed  in  Table  2.  The  4  levels  for 
threat  include  a  7.62mm  round  fired  at  45°  and  90®  obliquity  on  the  sidewall,  a  small  fragment 
simulator,  and  a  large  fragment  simulator.  Note  that  only  2  tire  position  levels,  drive  or  steer¬ 
ing,  are  considered.  The  case  in  which  two  tires  are  damaged,  requiring  twice  as  many  sam¬ 
ples,  is  handled  only  in  a  limited  sense.  Imbedded  in  the  stage  1  factorial  design  are  four 
treatment  combinations  having  two  damaged  tires  which  arise  from  a  1/2  replicate  of  a  24 
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Table  2.  1/2  Replication  of  4  x  2  =  32  Observations  -«-  4  Observations  for  2  Tires  Damaged. 


design.  The  remaining  4  observations  are  already  included  in  the  principal  stage  1  design. 
The  other  three  factors  are  handled  as  previously  noted.  This  design  allows  hypothesis  tests 
on  all  main  effects  and  on  most  first-order  interactions.  Due  to  the  anticipated  complexity  of 
the  relationship  among  variables  some  first-order  interactions  can  be  sacrificed  to  the  experi¬ 
mental  error  formed  by  the  remaining  second,  and  above,  order  interactions.  The  remaining 
variables  are  addressed  in  stage  1  as  suggested  in  Figure  2. 

Randomization  for  the  stage  1  design  is  complete  except  in  the  case  of  driving  team, 
vehicle,  and  terrain.  In  consideration  of  the  procedures  for  running  a  test  and  the  ease  in  which 
variables  can  be  changed  and  the  details  of  the  physical  set  up  some  compromises  were  made. 
The  complete  randomization  of  the  driving  teams  is  not  possible  because  both  teams  are  to 
be  used  simultaneously.  The  first  driver  in  the  rotation  for  each  team  was  randomized.  As 
indicted  in  Figure  2,  four  vehicles  are  used  but  are  not  included  as  test  factors.  To  mitigate 
their  effect  on  the  outcome,  they  have  been  selected  according  to  age  and  state  of  repair  and 
have  been  partially  randomized  over  the  design.  Also  noted  in  Figure  2,  the  three  terrains 
mentioned  in  Section  2.1  comprise  the  test  track.  The  course  layout  attempts  to  mix  or  ran¬ 
domize  the  terrains  so  that  not  all  the  mileage  for  one  type  will  be  traveled  before  the  next  is 
encountered. 

3.  CRITIQUE  OF  APPROACH 

In  this  section  we  address  the  primary  advantages  and  disadvantages  of  the  test  plan 
interpreted  in  terms  of  the  stated  military  and  experimental  objectives  and  follow  with  some 
comments  pertaining  to  the  consulting  technique  employed.  Beginning  with  the  military 
objectives,  all  of  the  variables  considered  important  by  TACOM  or  the  NATO  Finable  Stan- 
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Figure  2.  Incorporating  Remaining  Variables  in  the  Design. 


dards  are  included  in  the  test  plan  in  a  manner  suitable  to  TACOM.  Sometimes  this  requires 
compromise,  such  as  in  the  use  of  terraia  Terrain  is  considered  only  through  its  inclusion  in 
the  test  course  in  proportions  consistent  with  the  HMMWV  mission  profile.  For  some  other 
variables  the  military  interests  are  clarified  in  the  test  plan.  For  example,  normal  operating 
speeds  in  the  failure  definition  are  more  sensibly  tied  to  the  normal  performance  of  individual 
driving  teams.  Also,  efforts  to  handle  the  fragment  threat  result  in  a  reasonable  fragment 
simulation  procedure  which  may  be  used  in  follow-on  acceptance  testing. 

With  regard  to  experimental  objectives,  the  selection  of  STS  prototypes  for  further 
research  and  development  follows  directly  from  analysis  of  the  second  testing  stage.  Further, 
the  stage  1  plan  imposes  an  anolyzable  design  structure  on  a  complex  problem  providing  for 
the  testing  of  all  important  hypotheses.  In  addition,  running  the  experiment  in  stages  has  the 
emphasis  and  resource  advantages  mentioned  in  Section  2.2.  However,  the  test  plan  has 
several  weaknesses.  By  examining  the  standard  tire  only  in  stage  1,  comparisons  between  it 
and  other  STS  prototypes  are  hindered.  Experimental  error  is  an  issue  since  complete  ran¬ 
domization  is  not  possible  and  since  some  pooling  of  low-order  interactions  into  the  error 
term  may  be  necessary.  Choice  of  an  error  term  for  the  imbedded  test  of  the  two-tire  effect 
is  far  from  straight-forward,  particularly  since  four  of  the  eight  observations  must  be  used  in 
the  analysis  twice.  Finally,  we  had  to  take  some  liberties  in  the  combination  of  variables  to 
form  factors  so  that  a  design  would  be  possible  with  the  available  samples. 

As  to  consulting,  we  cannot  prove  the  usefulness  of  Hahn’s  guidelines,  but  we  hope  that 
the  illustration  is  convincing.  Surely,  the  information  can  be  obtained  through  other  methods, 
but  the  imposed  structure  of  this  approach  facilitates  a  very  comprehensive  investigation.  In 
the  end,  all  methods  must  be  judged  by  the  experimental  strategies  which  they  help  to 
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develop,  but  their  performance  is  hopelessly  confounded  with  the  skills  of  the  consultant  using 
them.  Of  course  the  purposes  of  those  strategies  are  to  meet  both  application  objectives  and 
satisfy  statistical  theory.  Whether  this  strategy  satisfies  those  purposes,  and  if  not,  whether 
fault  lies  with  the  consultant,  the  approach,  or  the  problem  are  questions  left  for  the  reader  to 
decide. 
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APPENDIX 


terrain 

tire  position 
vehicle 

driver 

tire  motion 
threat  obliquity 

tire  technology 
fragment  shape 
fragment  size 
fragment  velocity 
road  temperature 

vehicle  load 
driving  speed 
distance  to  tire 

round 

miles  until  failure 
delivery  method 


•  the  three  driving  surfaces  listed  as  primary  road,  secon¬ 
dary  road,  and  cross  country.  Each  will  induce  different 
tire  stress  and  all  are  included  in  the  vehicle’s  mission 
profile. 

•  placement  of  the  damaged  tires.  In  testing,  the  right 
front,  right  rear,  or  both  may  be  degraded. 

-  High  Mobility  Multi-purpose  Wheeled  Vehicle 
(HMMWV).  Ibis  vehicle’s  tire  system  is  the  program’s 
focus.  The  individual  HMMWV  effect  is  an  issue. 

-  operator  of  the  vehicle.  The  influence  of  different  drivers 
should  be  accounted  for. 

-  the  state  of  the  tire,  either  static  or  rolling. 


•  six  Survivable  Tire  Systems  to  be  compared, 
partially  determines  the  nature  of  the  tear  or  puncture. 

•  partially  determines  the  nature  of  the  tear  or  puncture. 

-  partially  determines  the  nature  of  the  tear  or  puncture. 

effects  the  tire  vulnerability.  Previous  testing  revealed 
reduced  susceptibility  to  puncture  with  low  tire  pressures. 

influences  tire  stress. 

influences  tire  stress. 

-  refers  to  distance  traveled  by  fragment  or  small  caliber 
munition. 

threat  munition(s)  to  be  used. 

-  response. 

•  fragment  propelling  methods  considered. 
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#  shots  on  tire 


•  the  number  of  punctures  to  be  made  in  each  tire  to 
obtain  its  degraded  state. 

subjective  assessments  -  comments  solicited  from  drivers  regarding  handling  of 

vehicle  when  tires  are  in  normal  or  degraded  mode. 
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1.  Introduction.  The  classic  scatter  diagram  is  a  fundamental  tool  in  the  construction  of  a 
model  for  data.  It  allows  the  eye  to  detect  such  structures  in  data  as  linear  or  nonlinear  features, 
clustering,  outliers  and  the  like.  Unfortunately,  scatter  diagrams  do  not  generalise  readily  beyond  three 
dimensions.  For  this  reason,  the  problem  of  visually  representing  multivariate  data  is  a  difficult, 
largely  unsolved  one.  The  principal  difficulty,  of  course,  is  the  fact  that  while  a  data  vector  may  bo 
arbitrarily  high  dimensional,  say  n,  Cartesian  scatter  plots  may  only  eaaily  be  done  in  two  dimensions 
and,  with  computer  graphics  and  more  effort,  in  three  dimensions.  Alternative  multidimensional 
representations  have  been  proposed  by  several  authors  including  Chernoff  (1973),  Fionberg  (1979), 
Cleveland  and  McGill  (1984a)  and  Carr  et  al.  (1986). 

An  important  technique  based  on  the  use  of  motion  is  the  computer-based  kinematic  display 
yielding  the  Illusion  of  three  dimensional  scatter  diagrams.  This  technique  was  pioneered  by  Friedman 
and  Tukey  (1973)  and  Is  now  available  in  commercial  soft  ware' packages  (Donohoe’s  MacSpin  and 
Velleman’s  Data  Desk).  Coupled  with  easy  data  manipulation,  the  kinematic  display  techniques  have 
spawned  the  exploitation  of  such  methods  as  projection  pursuit  (Friedman  and  Tukey,  1974)  and  the 
grand  tour  (Asimov,  1986).  Clearly,  projection-baaed  techniques  lead  to  Important  insights  concerning 
data.  Nonetheless,  one  must  be  cautious  in  making  inferences  about  high  dimensional  data  structures 
based  on  projection  methods  alone.  It  would  be  highly  desireable  to  have  a  simultaneous 
representation  of  all  coordinates  of  a  data  vector  especially  if  the  representation  treated  all  components 
in  a  similar  manner.  The  cause  of  the  failure  of  the  standard  Cartesian  coordinate  representation  is  the 
requirement  for  orthogonal  coordinate  axes.  In  a  3-dlmensional  world,  it  is  difficult  to  represent  more 
than  three  orthogonal  coordinate  axes.  We  propose  to  give  up  the  orthogonality  requirement  and 
replace  the  standard  Cartesian  axes  with  a  set  of  n  parallel  axes. 


2.  Parallel  Coordinates.  We  propose  as  a  multivariate  data  analysis  tool  the  following 
representation.  In  place  of  a  scheme  trying  to  preserve  orthogonality  of  the  u-dlmensional  coordinate 
axes,  draw  them  as  parallel.  A  vector  (x1(  xa,  ,  xn)  is  plotted  by  plotting  xa  on  axis  1,  x3  on  axis  2 
and  so  on  through  xn  on  axis  n.  The  points  plotted  in  this  manner  are  joined  by  a  broken  lino.  Figure 
2.1  illustrates  two  points  (one  solid,  one  dashed)  plotted  In  parallel  coordinate  representation.  In  this 
illustration,  the  two  points  agree  in  the  fourth  coordinate.  The  principal  advantage  of  this  plotting 
device  Is  clear.  Each  vector  (x1(  xa,  ...  ,  xn)  Is  represented  in  a  planar  diagram  so  that  each  vector 
component  has  essentially  the  same  representation. 

The  parallel  coordinates  proposal  has  its  roots  in  a  number  of  sources.  Griffon  (1958)  considers 
a  2-dimensional  parallel  coordinate  type  device  as  a  method  for  graphically  computing  the  Kendall  tau 
correlation  coefficient.  Hartigan  (1975)  describes  the  “profiles  algorithm"  which  he  describes  at 
“histograms  on  each  variable  connected  between  variables  by  identifying  cases."  Although  he  does  not 
recommend  drawing  all  profiles,  a  profile  diagram  with  all  profiles  plotted  is  a  parallel  coordinate  plot. 
There  Is  however  far  more  mathematical  structure,  particularly  high  dimensional  structure,  to  the 
parallel  coordinate  diagram  than  Hartigan  exploits.  Inselberg  (1985)  originated  the  parallel  coordinate 

1This  research  was  sponsored  by  the  Army  Research  Office,  Contract  DAAL03-87-K-0087 
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representation  as  a  device  for  computational  geometry.  His  1985  paper  is  the  culmination  of  a  series  of 
technical  reports  dating  from  1981.  Finally  we  note  that  Diaconis  and  Friedman  (1983)  discuss  the  so- 
called  M  and  N  plots.  Their  special  case  of  a  1  and  1  plot  is  a  parallel  coordinate  plot  in  two 
dimensions.  Indeed,  the  1  and  1  plot  is  sometimes  called  a  before-and-nfter  plot  and  has  a  much  older 
history.  The  fundamental  theme  of  this  paper  is  that  the  transformation  from  Cartesian  coordinates  to 
parallel  coordinates  is  a  highly  structured  mathematical  transformation,  hence,  maps  mathematical 
objects  into  mathematical  objects.  Certain  of  these  can  be  given  highly  useful  statistical 
interpretations  so  that  this  representation  becomes  a  highly  useful  data  analysis  tool. 

3.  Parallel  Coordinate  Geometry.  The  parallel  coordinate  representation  enjoys  some  elegant 
duality  properties  with  the  usual  Cartesian  orthogonal  coordinate  representation.  Consider  a  line  L  in 
the  Cartesian  coordinate  plane  given  by  h  y=mx+b  and  consider  two  points  lying  on  that  line,  Bay 
(a,  ma+b)  and  (c,  mc-f  b).  For  simplicity  of  computation  we  consider  the  xy  Cartesian  axes  mapped 
Into  the  xy  parallel  axes  as  described  in  Figure  3.1.  We  superimpose  a  Cartesian  coordinate  axes  t,u  on 
the  xy  parallel  axes  so  that  the  y  parallel  axis  has  the  equation  u=l.  The  point  (a,  ma+b)  in  the  xy 
Cartesian  system  maps  into  the  line  joining  (a,  0)  to  (ma+b,  1)  In  the  tu  coordinate  axes.  Similarly, 
(c,  mc+b)  maps  into  the  line  joining  (c,  0)  to  (mc+b,  1).  It  is  a  straightforward  computation  to  show 
that  these  two  lines  intersect  at  a  point  (in  the  tu  plane)  given  by  Z:  (  b(l— m)"1,  (1—m)"1).  Notice 
that  this  point  in  the  parallel  coordinate  plot  depends  only  on  m  and  b  the  parameters  of  the  original 
line  in  the  Cartesian  plot.  Thus  I  is  the  dual  of  L  and  we  have  the  Interesting  duality  result  that 
points  in  Cartesian  coordinates  map  into  lines  in  parallel  coordinates  while  lines  in  Cartesian 
coordinates  map  into  points  in  parallel  coordinates. 

For  0  <  (1-m)'1  <  1,  m  is  negative  and  the  intersection  occurs  between  the  parallel 
coordinate  axes.  For  m=  —  1,  the  intersection  is  exactly  midway.  A  ready  statistical  interpretation 
can  be  given.  For  highly  negatively  correlated  pairs,  the  dual  line  segments  in  parallel  coordinates  will 
tend  to  cross  near  a  single  point  between  the  two  parallel  coordinate  axes.  The  scale  of  one  of  the 
variables  may  be  transformed  in  such  a  way  that  the  intersection  occurs  midway  between  the  two 
parallel  coordinate  axes  in  which  case  the  slope  of  the  linear  relationship  is  negative  oiie. 

In  the  case  that  (l~m)_1<0  or  (1— m)*1>l,  m  Is  positive  and  the  intersection  occurs  external 
to  the  region  between  the  two  parallel  axes.  In  the  epeclal  case  mssl,  this  formulation  breaks  down. 
However,  it  is  clear  that  the  point  pain  are  (a,  a+b)  and  (c,  c+b).  The  dual  lines  to  these  points  arc 
the  lines  in  parallel  coordinate  space  with  slope  b'1  and  Intercepts  —  ab"1  and  —  cb'1  respectively.  Thus 
the  duals  of  these  lines  in  parallel  coordinate  space  are  parallel  lines  with  slope  b'1.  We  thuB  append 
the  ideal  points  to  the  parallel  coordinate  plane  to  obtain  a  projective  plane.  These  parallel  lines 
intersect  at  the  ideal  point  in  direction  b'1.  In  the  statistical  setting,  we  have  the  following 
Interpretation.  For  highly  positively  correlated  data,  we  will  tend  to  have  lines  not  intersecting 
betweeu  the  parallel  coordinate  axes.  By  suitable  linear  rescaling  of  one  of  the  variables,  the  lines  may 
be  made  approximately  parallel  in  direction  with  elope  b'1.  In  this  case  the  slope  of  the  linear 
relationship  between  the  rescaled  variables  is  one.  See  Figures  3.2  for  an  illustration  of  large  positive 
and  large  negative  correlations.  Of  course,  nonlinear  relationships  will  not  respond  to  simple  linear 
rescaling.  However,  by  suitable  nonlinear  transformations,  it  should  be  possible  to  transform  to 
linearity.  The  point-line,  line-point  duality  seen  In  the  transformation  from  Cartesian  to  parallel 
coordinates  extends  to  conic  sections.  An  instructive  computation  involves  computing  in  the  parallel 
coordinate  space  the  image  of  an  ellipse  which  turns  out  to  be  a  general  hyperbolic  form.  For  purposes 
of  conserving  space  we  do  not  provide  the  details  here. 

It  should  be  noted,  however,  that  the  solution  to  this  computation  is  not  a  locus  of  points,  but 
a  locus  of  lines,  e  line  conic.  The  envelope  of  this  line  conic  is  a  point  conic.  In  the  case  of  this 
computation,  the  point  conic  in  the  original  Cartesian  coordinate  plane  is  an  ellipse,  the  image  in  the 
parallel  coordinate  plane  is  as  we  have  just  seen  a  line  hyperbola  with  a  point  hyperbola  as  envelope. 
Indeed,  it  is  true  that  a  conic  will  always  map  into  a  conic  and,  in  particular,  an  ellipse  will  always 
map  into  a  hyperbola.  The  converse  Is  not  true.  Depending  on  the  details,  a  hyperbola  may  map  Into 
an  ellipse,  a  parabola  or  another  hyperbola.  A  fuller  discussion  of  projective  transformations  of  conics 
is  given  by  Dlmsdale.  (1984).  Inselberg  (1985)  generalises  this  notion  into  parallel  coordinates  resulting 
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in  what  he  calls  hstars. 

We  mentioned  the  duality  between  points  and  lines  and  conics  and  conics.  It  is  worthwhile  to 
point  out;  two  other  nice  dualities.  Rotations  in  Cartesian  coordinates  become  translations  in  parallel 
coordinates  and  vice  versa.  Perhaps  more  interesting  from  a  statistical  point  of  view  is  that  points  of 
inflection  in  Cartesian  space  become  cusps  in  parallel  coordinate  space  and  vice  versa.  Thus  the 
relatively  hard-to-detect  inflection  point  property  of  a  function  becomes  the  notably  more  easy  to 
defect  cusp  in  the  parallel  coordinate  representation.  Inselberg  (1085)  discusses  these  properties  in 
detail. 

4.  Further  Statiatiral  Interpretations.  Since  ellipses  map  Into  hyperbolas,  we  can  have  an  easy 
template  for  diagnosing  uncor  elated  data  pain.  Consider  Figure  3.9..  With  a  completely  uncorrelated 
data  set,  wo  would  expect  the  2-dimensional  scatter  diagram  to  fill  substantially  a  circumscribing 
circle.  As  illustrated  in  Figure  3.2,  the  parallel  coordinate  plot  would  approximate  a  figure  with  a 
hyperbolic  envelope.  As  the  correlation  approaches  negative  one,  the  hyperbolic  envelope  would  deepen 
so  that  in  the  limit  we  would  have  a  pencil  of  tines,  what  we  like  to  call  the  cross-over  effect.  As  the 
correlation  approaches  positive  one,  the  hyperbolic  envelope  would  widen  with  fewer  and  fewer  cross¬ 
overs  so  that  in  the  limit  we  would  have  parallel  lines.  Thus  correlation  structure  can  be  diagnosed 
from  the  parallel  coordinate  plot.  As  noted  earlier,  Griffen  (1958)  used  this  as  a  graphical  device  for 
computing  the  Kendall  tau. 

O/iffen,  In  fact,  attributes  the  graphical  device  to  Holmes  (1928)  which  predates  Kendall’s 
discussion.  The  computational  formula  is 


r  m  1  - 


H5&5 


where  X  is  the  number  of  intersections  resulting  by  connecting  the  Ira  rankings  of  each  member  by 
lines,  one  ranking  having  been  put  in  natural  order  While  the  original  formulation  was  framed  in 
terms  of  ranks  for  both  x  and  y  Axes,  it  is  clear  that  the  numbe.  of  crossings  is  invariant  to  any 
monotone  increasing  transformation  of  either  x  or  y,  the  ranks  being  one  such  transformation.  Because 
of  this  scale  invariance,  one  would  expect  rank-based  statistics  to  have  an  intimate  relationship  to 
parallel  coordinates. 

It  is  clear  that  if  there  is  a  perfect  positive  linear  relationship  with  no  crossings,  then  X  =  0 
and  r  =  1.  Similarly,  if  there  .  a  perfect  negative  linear  relationsuip,  Figure  3.2  is  again  appropriate 
and  we  have  a  pencil  of  lines.  Since  every  line  meets  every  other  line,  the  number  of  intersections  is 
(2  )  so  that 

r  =  1  -  -4-2^  =  -1. 
n(n-l) 

It  should  be  further  noted  that  clustering  is  easily  diagnosed  using  the  parallel  coot  .mate 
representation. 

So  far  we  have  focused  primarily  on  pairwise  parallel  coordinate  relationships.  The  idea 
however  is  that  we  can,  sc  to  speak,  stack  these  diagrams  and  represent  nil  n  dimensions 
simultaneously.  Figure  4.1  tr.us  illustrates  6-dimensinnal  Gaussian  uncorrelated  data  plotted  in 
parallel  coordinates.  A  6-dimensional  ellipsoid  would  have  a  similar  general  shape  but  with  aypcrbolaa 
of  different  depths.  This  data  is  deep  ocean  acoustic  noise  and  is  illustrative  of  what  might  be 
expected. 

Figure  4.2  is  illustrative  of  some  data  structures  one  might  see  in  a  five-dimensional  data  set. 
First  it  should  be  noted  that  the  plots  along  any  given  axis  represent  dot  diagrams  (a  refinement  of  the 
histograms  of  Hartigan),  hence  convey  graphically  the  one-dimensional  marginal  distributions.  In  this 
illustration,  the  first  axis  is  meant  to  have  an  approximately  normal  distribution  shape  while  axis  two 
the  shape  of  the  negative  of  a  As  discussed  above,  the  pairwise  comparisons  can  be  made.  Figure 
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4.2  illustrates  a  number  of  instances  of  linear  (both  negative  and  positve),  nonlinear  and  clustering 
situations.  Indeed,  it  is  clear  that  there  is  a  3-dimensional  cluster  along  coordinates  3,  4  and  4. 

Consider  also  the  appearance  of  a  mode  in  parallel  coordinates.  The  mode  is,  intuitively 
speaking,  the  location  of  the  most  intense  concentration  of  probability.  Hence,  in  a  sampling  situation 
it  will  be  the  location  of  the  most  intense  concentration  of  observations.  Since  observations  are 
represented  by  broken  line  segments,  the  mode  in  parallel  coordinates  will  be  represented  by  the  most 
intense  bundle  of  broken  line  paths  in  the  parallel  coordinate  diagram.  Roughly  speaking,  we  should 
look  for  the  most  intense  flow  through  the  diagram.  In  Figure  4.2,  such  a  flow  begins  near  the  center 
of  coordinate  axis  one  and  finishes  on  the  left-hand  side  of  axis  five. 

Figure  4.2  thus  illustrates  some  data  analysis  features  of  the  parallel  coordinate  representation 
including  the  ability  to  diagnose  one-dimensional  features  (marginal  densities),  two-dimensional 
features  (correlations  and  nonlinear  structures),  three-dimensional  features  (clustering)  and  a  five- 
dimensional  feature  (the  mode).  In  the  next  section  of  this  paper  we  consider  a  real  data  set  which  will 
be  illustrative  of  some  additional  capabilities. 

5.  An  Auio  Data  Example.  We  illustrate  parallel  cooordinatee  as  an  exploratory  analysis  tool 
on  data  about  86  1980  model  year  automobiles.  They  consist  of  price,  miles  per  gallon,  gear  ratio, 
weight  and  cubic  inch  displacement.  For  n  =  5,  3  presentations  are  needed  to  present  all  pairwise 
permutations.  Figures  5.1,  5.2  and  5.3  are  these  three  presentations.  In  Figure  5.1,  perhaps  the  moBt 
striking  feature  is  the  cross-over  effect  evident  in  the  relationship  between  gear  ratio  and  weight.  This 
suggests  a  negative  correlation.  Indeed,  this  is  reasonable  since  a  heavy  car  would  tend  to  have  a  large 
engine  providing  considerable  torque  thus  requiring  a  lower  gear  ratio.  Conversely,  a  light  car  would 
tend  to  have  a  small  engine  providing  small  amounts  of  torque  thus  requiring  a  higher  gear  ratio. 

Consider  as  well  the  relationship  between  weight  and  cubic  inch  displacement.  In  this  diagram 
we  have  a  considerable  amount  of  approximate  parallelism  (relatively  few  crossings)  suggesting  positive 
correlation.  This  is  a  graphic  representation  of  the  fact  that  big  cars  tend  to  have  big  engines,  a  fact 
moat  are  prepared  to  believe.  Quite  striking  however  is  the  negative  slope  going  from  low  weight  to 
moderate  cubic  inch  displacement.  This  is  clearly  an  outlier  which  is  unusual  in  neither  variable  but  in 
their  joint  relationship.  The  same  observation  is  highlighted  in  Figure  5.2. 

The  relationship  between  miles  per  gallon  and  price  is  also  perhaps  worthy  of  comment.  The 
left-hand  side  shows  an  approximate  hyperbolic  boundary  while  the  right-hand  side  clearly  illustrates 
the  cross-over  effect.  This  suggests  for  inexpensive  cars  or  poor  mileage  cars  there  is  relatively  little 
correlation.  However,  costly  cars  almost  always  get  relatively  poor  mileage  while  good  gas  mileage  cars 
are  almost  always  relatively  inexpensive. 

Turning  to  Figure  5.2,  the  relationship  between  gear  ratio  and  miles  per  gallon  is  instructive. 
This  diagram  is  suggestive  of  two  classes.  Notice  that  there  are  a  number  of  observations  represented 
by  line  segments  tilted  slightly  to  the  right  of  vertical  (high  positive  slope)  and  a  somewhat  larger 
number  with  a  negative  slope  of  about  —1.  Within  each  of  these  two  classes  we  have  approximate 
parallelism.  This  suggests  that  the  relationship  between  gear  ratios  and  miles  per  gallon  is 
approximately  linear,  a  believable  conjecture  since  low  gears  =  big  engines  =  poor  mileage  while  high 
gears  =  small  engines  =  good  mileage.  What  is  intriguing,  however,  is  that  there  seems  to  be  really 
two  distinct  classes  of  automobiles  each  exhibiting  a  linear  relationship,  but  with  different  linear 
relationships  within  each  class. 

Indeed  in  Figure  5.3,  the  third  permutation,  we  are  able  to  highlight  this  separation  into  two 
classes  in  a  truly  5-dimensional  sense.  The  shaded  region  in  Figure  5.3  describes  a  class  of  vehicles  with 
relatively  poor  gas  mileage,  relatively  heavy,  relatively  inexpensive,  relatively  large  engines  and 
relatively  low  gear  ratios.  Figure  5.4  is  a  repeat  of  this  graphic  but  with  different  shading  highlighting 
a  class  of  vehicles  with  relatively  good  gas  mileage,  relatively  light  weight,  relatively  inexpensive, 
relatively  small  engines  and  relatively  high  gear  ratios.  In  1980,  these  two  characterizations  describe 
respectively  domestic  automobiles  and  imported  automobiles. 

6.  Graphical  Extensions  of  Parallel  Coordinate  Plots.  The  basic  parallel  coordinate  idea 
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suggests  some  additional  plotting  devices.  We  rail  these  respectively  the  Parallel  Coordinate  Density 
Plots,  Relative  Slope  Plots  and  Color  Histograms.  These  are  extensions  of  the  basic  idea  of  parallel 
coordinates,  but  structured  to  exploit  additional  features  or  to  convey  certain  information  more  easily. 

6.1  Parallel  Coodinate  Density  Plots.  While  the  basic  parallel  coordinate  plot  Is  a  useful 
device  itself,  like  the  conventional  scatter  diagram,  it  suffers  from  heavy  overplotting  with  large  data 
sets.  In  order  to  get  around  this  problem,  we  use  a  parallel  coordinate  density  plot  which  is  computed 
as  follows.  Our  algorithm  is  based  on  the  Scott  (1085)  notion  of  average  shifted  histogram  (ASH)  but 
adapted  to  the  parallel  coordinate  context.  As  with  an  ordinary  two  dimensional  histogram,  we  decide 
on  appropriate  rectangular  bins.  A  potential  difficulty  arises  because  a  line  segment  representing  a 
point  may  appear  in  two  or  more  bins  in  the  Bame  horisontal  slice.  Obviously  if  we  have  k  n- 
dimenuional  observations,  we  would  like  to  form  a  histogram  based  on  k  entries.  However,  since  the 
line  segment  could  appear  in  two  or  more  bins  in  a  horisontal  slice,  the  count  for  any  given  horizontal 
slice  is  at  least  k  and  may  be  bigger.  Moreover,  every  horisontal  slice  may  not  have  the  same  count. 
To  get  around  this,  we  convert  line  segments  to  points  by  intersecting  each  line  Segment  with  a 
horizontal  line  passing  through  the  middle  of  the  bin.  This  gives  us  an  exact  count  of  It  for  each 
horizontal  slice.  We  construct  an  ASH  for  each  horizontal  slice  (typically  averaging  5  histograms  to 
form  our  ASH).  We  have  used  contours  to  represent  the  two-dimensional  density  although  gray  scale 
shading  could  be  used  in  a  display  with  sufficient  bit-plane  memory.  An  example  of  an  parallel 
coordinate  density  plot  is  given  in  Figure  6.1.  Parallel  coodinate  density  plots  have  the  advantage  of 
being  graphical  representations  of  data  setc  which  are  simultaneously  high  dimensional  and  very  large. 

6.2  Relative  Slope  Plots.  We  have  already  seen  that  parallel  line  segments  in  a  parallel 
coordinate  plot  correspond  to  high  positive  correlatlou  (linear  relationship).  As  in  our  automobile 
example,  it  is  possible  for  two  or  more  sets  of  liuear  relationships  to  exist  simultaneously,  in  an 
ordinary  parallel  coordinate  plot,  we  see  these  as  seta  of  parallel  lines  with  distinct  slopes.  The  work  of 
Cleveland  and  McGill  (1984b)  suggests  that  comparison  of  slopes  (angles)  is  a  relatively  inaccurate 
judgement  task  and  that  it  Is  much  easier  to  comparo  magnitudes  on  the  same  scale.  The  relative 
slope  plot  is  motivated  by  this.  In  an '  n-dimenaional  relative  slope  plot  there  are  u— 1  parallel  axes, 
each  corresponding  to  a  pair  of  axes,  say  x(  and  xy,  with  xj  regarded  as  the  lower  of  the  two  coordinate 
axes.  For  each  observation,  the  slope  of  the  line  segment  between  the  pair  of  axes  is  plotted  as  a 
magnitude  between  —1  and  +1.  The  maximum  positive  slope  is  coded  as  +1,  the  minimum  negative 
slope  as  —  1  and  a  slope  of  oo  as  0.  The  magnitude  is  calculated  ss  coe  7  where  tf  is  the  angle  between 
the  Xj  axis  and  the  line  segment  corresponding  to  the  observation.  Each  individual  observation  in  the 
relative  slope  plot  corresponds  to  a  vertical  section  tiirough  the  axis  system.  An  example  of  a  relative 
slope  plot  is  given  in  Figure  6.2.  Notice  that  since  slopes  are  coded  as  heights,  simply  laying  a 
straightedge  will  allow  us  to  discover  sets  of  linear  relationships  within  the  pair  of  variables  x(  and  Xj. 

6.3  Color  Histograms.  The  basic  set-up  for  the  color  histogram  is  similar  to  the  relative  slope 
plots.  For  an  n-dimenaional  data  set,  there  are  n  parallel  axee.  A  vertical  section  through  the  diagram 
corresponds  to  an  observation.  The  idea  is  to  code  the  magnitude  of  an  observation  along  a  given  axis 
by  a  color  bin,  the  colors  being  chosen  to  form  a  color  gradient.  We  typically  choose  8  to  16  colors. 
The  diagram  is  drawn  by  choosing  an  axis,  say  xk,  and  sorting  the  observations  in  ascending  order. 
Along  this  axis,  we  see  blocks  of  color  arranged  according  to  the  color  gradient  with  the  width  of  the 
block  being  proportional  to  the  number  of  observations  falling  into  the  color  bin.  The  observations  on 
the  other  axes  are  arranged  in  the  order  corresponding  to  the  xk  axis  and  color  coded  according  to  their 
magnitude.  Of  course,  if  the  same  color  gradient  shows  up  say  on  the  xm  axis  as  on  the  xk,  then  we 
know  x4  is  positively  “correlated”  with  xm.  If  the  color  gradient  is  reversed,  we  know  the  “correlation” 
is  negative.  We  used  the  phrase  “correlation”  advisedly  since  in  fact  if  the  color  gradient  is  the  same 
but  the  color  block  sizes  are  diffeient,  the  relationship  is  nonlinear.  Of  course  if  the  xm  axis  dhows 
color  speckle,  there  is  no  “correlation”  and  xk  is  unrelated  to  xm>  An  example  of  a  color  histogram  !b 
given  in  Figure  6.3  (for  purposed  of  reproduction  here  it  is  really  a  gray-scale  histogram). 

7.  Implementations  and  Experiences.  Our  parallel  coordinates  data  analysis  software  has  been 
implemented  in  two  forms,  one  a  PASCAL  program  operating  on  the  IBM  RT'  under  the  AIX 
operating  system.  This  code  allows  for  up  to  four  simultaneous  windows  and  offers  simultaneous 
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display  of  parallel  coordinates  and  scatter  diagram  displays.  It  offers  highlighting,  rooming  and  other 
similar  features  and  also  allows  the  possibility  of  nonlinear  rescaling  of  each  axis.  It  incorporates  axes 
permutations  and  also  includes  Parallel  Coordinate  Density  Plots,  Relative  Slope  Plots  and  Color 
Histograms. 

Our  second  implementation  is  under  development  in  PASCAL  for  MS-DOS  machines  and 
includes  similar  features.  In  addition,  it  has  a  mouse-driven  painting  capability  and  can  do  real-time 
rotation  of  3-dimensional  scatterplots.  Both  programs  use  EGA  graphics  standards,  with  the  second 
also  using  VGA  or  Hercules  monochrome  standards. 

We  regard  the  parallel  coordinate  representation  as  a  device  complementary  to  scatterplots.  A 
major  advantage  of  the  parallel  coordinate  representation  over  the  scatterplot  matrix  is  the  linkage 
provided  by  connecting  points  on  the  axes.  This  linkage  is  difficult  to  duplicate  in  the  scatterplot 
matrix.  Because  of  the  projective  line-point  duality,  the  structures  seen  In  a  scatterplot  can  also  be 
seen  in  a  parallel  coordinate  plot.  Moreover,  the  work  of  Cleveland  and  McGill  (1984b)  suggests  that 
it  is  easier  and  more  accurate  to  compare  observations  on  a  common  scale.  The  parallel  coordinate 
plot  and  the  derivatives  of  It  de  facto  have  a  common  scale  and  so  for  example  a  sense  of  variability 
and  central  tendency  among  the  variables  are  easier  to  grasp  visually  In  parallel  coordinates  when 
compared  with  the  scatterplot  matrix.  On  the  other  hand,  one  might  interpret  all  the  ink  generated  by 
the  lines  as  a  significant  disadvantage  of  the  parallel  coordinate  plot.  Our  experience  on  this  is  mixed. 
Certainly  for  large  data  sets  on  hard  copy  this  is  a  problem.  When  viewed  on  an  interactive  graphics 
screen  particularly  a  high  resolution  screen,  we  have  often  found  that  individual  points  in  a  scatterplot 
can  get  loet  because  they  are  simply  not  bright  enough.  That  does  cot  happen  iu  a  parallel  coordinate 
plot.  However,  If  many  points  are  plotted  in  monochrome,  It  is  hard  to  distinguish  between  points. 
We  have  gotten  around  this  problem  by  plotting  distinct  joints  in  different  colors.  In  an  EGA 
implementation,  this  means  16  colors.  This  Is  surprisingly  effective  in  separating  points.  In  one 
experiment,  we  plotted  5000  5-dimensional  random  vectors  using  19  colors,  and  Inspite  of  total 
overplotting,  we  were  still  able  to  see  some  structure.  In  data  seta  of  somewhat  smaller  scale,  wt  have 
Implement  a  scintillation  technique.  With  this  technique,  when  there  Is  overplotting  we  cause  tie 
screen  view  to  scintillate  between  the  colors  representing  the  overplotted  points.  The  speed  of 
scintillation  is  is  proportional  to  the  number  of  points  overplotted  and  by  carefully  tracing  colors,  one 
can  follow  an  Individual,  point  through  the  entire  diagram. 

We  have  found  painting  to  be  an  extraordinarily  effective  technique  in  parallel  coordinates. 
We  have  a  painting  scheme  that  not  only  paints  all  lines  within  a  given  rectangular  area,  but  also  all 
line  lying  between  to  slope  constraints.  This  is  very  effective  in  separating  clusters.  We  also  use 
invisible  paint  to  eliminate  observation  points  from  the  data  set  temporarily.  This  is  a  natural  way  of 
doing  a  subset  selection. 
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Figure  4.1a  Parallel  coordinate  plot  of  a  circle. 


Figure  4.1b  Parallel  coordinate  plot  of  6  channel  aonar  data. 

Th«  data  la  uncorrelated  Gauaaian  noiaa.  The  ancond 
coordinate  rapreaenta  a  relatively  remote  hydrophone  and  hu  a 
aomawhat  different  mean.  Notice  the  approximate  hyperbolic 
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Figure  4. 2  A  Ova  dimensional  scat  Ur  diagram  In  parallel 
coordlnaUa  illustrating  densities,  correlaiious,  three 

dimeneional  clustering  and  a  Ova  dimensional  mode. 
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Figure  5.1  A  parallel  coordinate  plot  La  five  dlm«ntiou  of 
automobile  data.  Nota  the  negative  correlation  between  gear 
ratios  and  weight. 
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Figure  5.2  The  second  permutation  of  the  five  dimensional 
presentation  of  the  automobile  data.  Notice  the  two  classes  of 
linear  relations  gear  ratio  and  miles  per  gallon. 
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Figure  6.1  Parallel  coordinate  denaity  plot  of  5000  uniform 
random  variables.  Thl*  plot  ha*  five  contour 
level*  5%,  25%,  50  %,  75%  and  95%. 


Figure  6.3  Color  hletogram  of  13  dimenaional  automobile  data. 
Thia  plot  ia  ataow  in  grey  acale  for  purpose*  of  reproduction. 
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COMPUTATIONAL  AND  STATISTICAL  ISSUES 
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Abstract 

Discrete-event  simulation  is  one  of  the  most  important  techniques  available  for  study¬ 
ing  complex  stochastic  systems.  In  this  paper  we  review  the  principal  methods  available 
for  analyzing  both  the  transient  and  steady-state  simulation  problems  in  sequential  and 
parallel  computing  environments.  Next  we  discuss  several  of  the  variance  reduction  meth¬ 
ods  designed  to  make  simulations  run  more  efficiently.  Finally,  a  short  discussion  is  given 
of  the  methods  available  to  study  system  optimization  using  simulation. 

Keywords:  stochastic  simulation,  output  analysis,  variance  reduction,  parallel  computa¬ 
tion,  and  system  optimization. 
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1.  Introduction. 

Computer  simulation  of  complex  otoch&dtic  systems  is  an  important  technique  for 
evaluating  system  performance.  The  starting  point  for  this  method  is  to  formulate  the 
time  varying  behavior  of  the  system  as  a  basic  stochastic  process  Y  s  {Y{t)  :  t  >  0}, 
where  y(-)  may  be  vector* valued.  [Discrete  time  processes  cau  also  be  handled.]  Next 
a  computer  program  is  written  to  generate  sample  realizations  of  V .  Simulation  output 
is  then  obtained  by  running  this  program.  Our  discussion  in  this  paper  is  centered  on 
the  analysis  of  this  simulation  output.  The  goal  being  to  develop  sound  probabilistic  and 
statistical  methods  for  estimating  system  performance. 

Two  principal  problems  arise:  the  transient  simulation  problem  and  the  steady*state 
simulation  problem.  Let  T  denote  a  stopping  time  and  X  m  h{Y(i) :  0  £  t  <  T},  where  h 
is  a  given  real* valued  function.  The  transient  problem  is  to  estimate  a  s  Examples 

of  o  include  the  following: 

«  -  TO •)}, 

/(rW)  4 

and 

a  m  P{Y  does  not  enter  A  before  to}- 

Here  to  is  a  fixed  time  (>  0),  /  is  a  given  re*i-valued  function,  and  A  is  a  given  subset  of 
the  state-space  of  Y.  The  tracsieut  problem  is  relevant  for  systems  running  for  a  limited 
(but  possibly  random)  length  of  time  that  cannot  be  expected  to  reach  a  steady-state.  Our 
goal  here  is  to  provide  both  point  and  interval  estimates  for  a. 

For  the  steady-state  problem  we  assume  the  Y  process  is  asymptotically  stationary 
in  the  sense  that 

\  -  a 

as  t  -4  oo.  Here  denotes  weak  convergence  and  f  is  a  given  real- valued  function 
defined  on  the  state-space  of  Y,  The  easiest  example  to  think  about  here  is  an  irreducible, 
positive  recurrent,  continuous  time  Markov  chain.  In  this  case  V'(t)  »►  Y  as  i  oo  and 
a  ~  E{f(Y)}.  Examples  of  a  in  this  case  include  the  following: 

a  ==  E{Yk)  (when  Y  is  real-valued), 
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a  »  P{Y  €  A}, 
and 

a  -  £MV)}, 

where  c  is  a  given  coat  function.  Again  ai  in  the  transient  case,  we  wish  to  construct  both 
point  and  interval  estimates  for  a. 

2.  Transient  Problem. 

Assume  we  have  a  computational  budget  of  t  time  units  with  which  to  simulate  the 
process  Y  and  estimate  a  a  E{X },  as  defined  in  Section  1.  In  a  sequential  computing 
environment  we  would  generate  independent,  identically  distributed  (lid)  copies 

where  the  X<’s  are  copies  of  X  and  r<  is  the  computer  time  required  to  generate  X,.  Let 
N(t)  denote  the  number  of  copies  of  X  generated  in  time  t;  this  is  just  the  renewal  process 
associated  with  the  iid  r/s.  A  natural  point  estimator  for  m  is 

{m 

,  m> o 

0  ,"1  ,  JV(t)  m  0. 

The  standard  asymptotic  results  for  Xnw  are  the  strong  law  of  large  numbers  (SLLN) 
and  the  central  limit  theorem  (CLT). 

STRONG  LAW  OP  LARGE  NUMBERS.  If  £{rj}  <  oo  and  £{|  Xx  |)  <  oo,  then 
%N(t)  -*♦’  «  a.J.  as  t  -»  oo. 

CENTRAL  LIMIT  THEOREM.  If  £{r*}  <  oc  and  var^}  <  oo,  then 

-  a)  ■*  (£{r,}  •  vufX.})1'8  ■  JV(0,1), 

where  N( 0, 1)  is  a  mean  zero,  variance  one  normal  random  variable.  The  SLLN  follows 
from  the  SLLN  for  iid  summands  and  the  SLLN  for  renewal  processes.  The  CLT  result 
can  be  found  in  BILLINGSLEY  (196S),  Section  17. 
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From  the  SLLN  we  see  that  Xs{t)  is  a  strongly  consistent  point  estimator  for  a.  Thus 
for  large  t  we  would  use  Xs(t)  as  our  point  estimate.  On  the  other  hand,  the  CLT  can  be 
used  in  the  standard  manner  to  construct  a  confidence  interval  for  a.  Here  the  constant 
J£{n}  •  var{Xi}  appearing  in  the  CLT  would  have  to  be  estimated. 

Suppose  now  that  we  are  in  a  parallel  computing  environment  with  p  independent 

processors.  Now  we  wish  to  estimate  a  for  a  fixed  t  as  p  — ►  00.  On  the  p  processors  we 

generate  iid  copies  of  (X,  r): 

(■X’uirtiM-Xia.i’ia)  ,**•,  (X\N\(ibriNx(i)) 

(A‘3i,rai),(A'ja,ra2)  (X2St(t)>r iNt(t)) 


(Xpu  fyi)t  (Xp2,Tp2)  (XpN,{  //,(<))» 

A  number  of  estimators  have  been  proposed  for  estimating  a  m  The  most  natural 

estimator  to  consider  first  is  that  obtained  by  averaging  the  realizations  of  X  across  each 
processor  and  then  averaging  these  sample  means.  This  leads  to 


where 


rgw  .  j  ids  g  **  -  w»® 


U  ,  Ni(t)  -  0. 

Here  the  processing  ends  on  all  processors  at  time  Tp  —  t,  If  f?{ri}  <  00  and  JS?{|  X  |}  <  00, 
then  for  all  t  >  0 

<*1  (p»*)  -*  a,s' 


as  p  -4  00.  Here  lx  is  the  indicator  function  of  the  set  A.  Unfortunately,  £{A}  # 
•  1  {r^o  }  e&d  so  ai(p,<)  is  not  strongly  consistent  for  a  as  p  ->  00. 

The  next  estimator  for  a  was  proposed  by  HEIDELBERGER  (1987).  For  this  esti¬ 
mator  we  let  all  processors  complete  the  replication  in  process  at  time  t.  The  estimator 


is 


«a(p»0  ■ 


P  NtW+l 


E  E  ** 


£|W(<)  + 1] 


i«l 


268 


Here  all  processuri  complete  by  time 


T,  ■  +  Tii  +  •  *  •  +  Twm+i]. 

Unfortunately,  Tp  —►  +oo  a.s.  as  p  -4  00.  However,  oa(p,  t)  is  strongly  consistent  for  a, 
To  see  this,  note  that  if  £7{|  X  |}  <  00  and  P{r  >  0}  >  0,  then  as  p  — ►  00 

{*<<0+1 

£  *« 

£(^'))  +  1/  -*W  «• 

The  equality  about  is  simply  Wald’s  equation.  Finally,  since  aa(p,  t)  is  a  ratio  estimator, 
a  CLT  is  also  available  from  which  a  confidence  interval  can  be  constructed. 

The  last  estimator  we  consider  was  proposed  by  HEIDELBERGER  and  GLYNN 
(1987).  Here  we  set 

<*s(P»0  ■ 

*  <-1 

where 


Given  N(t)  >  1,  Heidelberger  and  Glynn  show  that  the  pairs  of  random  variables  (Xu  n), 
. . . ,  r>/(t))  are  exchangeable.  Using  this  fact,  they  prove  that  £{.£jv<(()}  *  E{X x}. 

Since  the  jfjy^’s  are  iid,  we  see  that  03 (t)  is  strongly  consistent  for  a  »■  E{X\ }.  Since 
the  summands  in  aj(p,t)  are  iid,  the  standard  CLT  holds  (under  appropriate  variance 
assumptions)  and  can  be  used  to  develop  a  confidence  interval  for  a.  Note  that  the 
definition  of  requires  the  tth  processor  to  complete  the  replication  in  process  at 

time  t,  if  no  observations  have  been  completed  by  time  t;  i.e.,  ru  >  t.  Thus  the  completion 
time  for  all  p  processors  is  given  by 


Tp  m  {max(t,ftt)}. 

While  Tp  -»  00  o.j.  as  p  -*  00  (if  P{rn  >  t}  >  0),  Tp  goes  to  infinity  at  a  much  slower 
rate  than  is  the  case  for  aia(p,  t).  They  also  show  that  the  following  CLT  holds: 
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aa  t  -4  oo,  where  we  assume  0  <  <r2  ■  uar{Xi}  <  oo  and  0  <  E{t\ }  <  oo.  Thus 
can  also  be  used  in  a  sequential  environment  to  estimate  a. 

3.  Steady-State  Problem. 

The  steady-state  estimation  problem  is  considerably  more  difficult  than  the  transient 
estimation  problem.  This  difficulty  stems  from  the  following  considerations:  (i)  need  to 
estimate  long-run  system  behavior  from  a  finite  length  simulation  run;  (ii)  an  initial  bias 
(or  transient)  usually  is  present  since  the  process  being  simulated  is  non-stationary;  and 
(iii)  strong  autocorrelations  are  usually  present  in  the  process  being  simulated.  While 
classical  statistical  methods  can  often  be  used  for  the  transient  estimation  problem,  these 
methods  generally  fail  for  the  steady-state  estimation  problem  for  the  reasons  mentioned 
above. 

Assume  our  simulation  output  process  is  Y  m  {Y(t)  :  t  >  0}  and  for  a  given  real¬ 
valued  function  / 

ctt)  a  \  J*  +  a.  (1) 

As  stated  above,  we  wish  to  construct  point  and  interval  estimators  for  o.  In  addition  to 
(1),  many  methods  also  assume  that  a  positive  constant  o  exists  such  that  the  following 
CLT  holds: 

-<*]**•  JV( 0, 1)  (2) 

as  t  -4  oo.  from  (1)  and  (2)  we  can  construct  a  point  estimate  and  confidence  interval  for 
a  provided  we  can  estimate  a.  Estimating  a  is  generally  the  hardest  problem. 

A  variety  of  methods  have  been  developed  to  address  the  steady-state  estimation 
problem.  In  Figure  1  we  have  given  a  break-down  of  these  methods.  Most  of  the  methods 
are  single  replicate  methods,  since  multiple  replicate  methods  tend  to  be  inefficient  because 
of  the  initial  bias  problem. 

Here  we  only  consider  single  replicate  methods.  These  methods  are  of  two  types: 
those  that  consistently  estimate  a  and  those  in  which  a  is  cancelled  out. 

For  consistent  estimation  of  <r,  we  need  a  process  {j(t)  :  t  >  0}  such  that  s(t)  =>  <r. 
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In  which  case  (2)  leads  to  a  100(1  -  6)  %  confidence  interval  for  a  given  by 

t«(‘)  -  i(l  -  i/2Xt)/tl/i,a(t)  + 1(1  -S/iXtyt1'*], 

where  $(*(1  -  6/2))  *  1  -  6/2  and  $  is  the  standard  normal  distribution  function. 

On  the  other  hand,  the  canceling  out  methods  require  a  non-vanishing  process  {Z(i) : 
t  >  0}  such  that 

[<‘^(o(i)-<.),Z(<)!*[<rW(0,l),<TZ] 

as  t  — ►  oo.  Then  using  the  continuous  mapping  theorem  (cf.,  BILLINGSLEY  (1968),  p. 
30)  we  have 

«1'J(a(t)-a)/Z(()-.tf(0,l)/Z  (3) 

as  t  — ►  oo.  Note  from  (3)  that  a  has  been  cancelled  out  in  a  manner  reminiscent  of  the 
t-statistic. 

First  we  discuss  one  of  the  methods  in  which  t r  is  consistently  estimated,  namely, 
the  regenerative  method;  see  IGLEHART  (1978)  for  a  discussion  of  this  method  plus 
other  background  material.  Here  we  assume  that  the  simulation  output  process  Y  is  a 
regenerative  process.  We  are  given  a  real-valued  function  /  and  wish  to  estimate  a(f)  a 
E{f(Y)},  where  Y(t)  Y  as  t  -►  oo.  Again  it  is  convenient  to  think  of  Y  as  an 
irreducible,  positive  recurrent,  continuous  time  Markov  chain.  Let  T(0)  ■  0,T\,  r3) ...  be 
the  regeneration  times  for  Y  and  set  -  T<-|,»  >  1.  The  r/a  are  the  lengths  of 

the  regenerative  cycles.  Next  define  the  areas  under  the  Y  process  in  the  Jfeth  regenerative 
cycle  by 

n(/)  -  r  f[Y(s)\ds. 

The  following  basic  facts  provide  the  foundation  for  the  regenerative  method: 

(i)  the  pairs  {(Y*(/),  r*)  :  k  >  1}  are  iid; 

(ii)  if  E{ |  f(Y)  |}  <  oo,  then  o (/)  -  E{YxU))/E{rxy 

The  regenerative  method  can  be  developed  on  either  the  intrinsic  time  seals  (t)  or  on  the 
random  time  scale  (n)  corresponding  to  the  number  of  regenerative  cycles  simulated.  On 
the  intrinsic  time  scale  our  point  estimate  for  a  is  given  by 

«((,/)■  m,))i,, 
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where  t  is  the  length  of  time  the  simulation  is  run.  On  the  random  time  scale  our  point  is 
given  by 

«»(/)  a  ?.(/)/*., 

where  ?„(/)  (respectively,  f„)  is  the  sample  mean  of  Yi(/), ...» Yn(f)  (ri, ...»  r„).  Here  the 
Y  process  is  simulated  to  the  completion  of  n  regenerative  cycles.  Using  the  basic  facts 
(i)  and  (ii)  above,  it  can  be  shown  that  both  a(t,/)  and  a „(/)  are  strongly  consistent  for 
<*(/)  as  t  and  n  respectively  tend  to  infinity.  Next  we  define  Z*  a  Y*(/)  -  o(/)r*  and 
assume  that  var{Z*}  s  <r3  <  oo.  Then  it  can  be  shown  that  the  following  two  CLT’s  hold 
as  t  oo  and  n  — » oo: 


*'/»(«<(,/)  -  <*(/)]  -  })JV(0, 1), 

and 

_  «(/)]  *  (,/JSfa  >)JV(0, 1). 

These  two  CLT’s  can  then  be  used  to  construct  confidence  intervals  for  at(f)  provided  both 
a2  and  £{ri}  can  be  estimated.  The  mean  E{+i}  is  -rily  estimated  by  fn  and  <r2  can  be 
estimated  from  its  definition  in  terms  of  Yi(f)  and  r\. 

Next  we  turn  to  a  discussion  of  the  principal  method  available  for  canceling  out  <r. 
This  is  the  method  of  standardized  time  serios  developed  by  SCHRUBGN  (1983).  Our 
discussion  is  based  on  the  paper  GLYNN  and  IGLEHART  (1989)  and  uses  some  results 
from  weak  convergence  theory;  see  BILLINGSLEY  (1968)  for  background  on  this  theory, 
from  our  output  process  Y  we  form  the  random  elements  of  C[0, 1],  the  space  of  real- valued 
continuous  functions  on  the  interval  (0, 1],  given  by 

n  J0 

and 

*„(<)  ■  n^lTnM  -  Of], 

where  0  <  t  ^  1  and  n  >  1.  Now  we  make  the  basic  assumption  that  a  finite,  positive 
constant  a  exists  such  that 

Xn  <rB  as  n  — ►  oo,  (4) 
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where  B  is  standard  Brownian  motion.  This  assumption  holds  for  a  wide  class  of  output 
processes.  To  find  the  scaling  process  {Z(t)  :  t  >  0}  consider  the  class  M  of  functions 
g  :  C(0, 1]  -+  ]R  such  that 

(i)  g(otx)  at  ag( i)  for  all  ot  >  0  and  x  €  C[0, 1]; 

(ii)  g(3)  >  0  with  probability  one; 

(iii)  g(x  +  0  k)  ■  g(x)  for  all  real  0  and  x  €  C[0, 1],  where  k(t)  «  t; 

(iv)  P{B  €  D(g))  »  0,  where  D(g)  is  the  set  of  discontinuities  of  g. 

The  process 

S»(0  *  0<(<l, 

is  called  a  standardized  time  series.  Using  weak  convergence  arguments  it  is  easy  to  shew 
from  (4)  that 

Sn(  1)  *  B(l)/g(JB)  (5) 

as  n  — *  oo.  Unfolding  this  CLT  we  have  the  following  100(1  -  S)%  confidence  interval  for 
a: 

(f.(i)  -  «a  -  «/2)ir(f»),  r,d) + «(</*)»(?.)], 

where  P{B(l)/g(B)  *(a)}  ■»  a  for  0  £  a  £  1.  Thus  each  jf  6  M  gives  rise  to  a 

confidence  interval  for  a  provided  we  can  find  the  distribution  of  B(\)/g(B).  Fortunately, 
this  can  be  done  for  a  number  of  interesting  g  functions. 

One  of  the  g  functions  leads  to  the  batch  means  method,  perhaps  the  most  popular 
method  for  steady-state  simulation.  We  conclude  our  discussion  of  the  method  of  stan¬ 
dardized  time  series  by  displaying  this  special  g  function.  To  this  end  we  first  define  the 
Brownian  bridge  mapping  V  :  C[0, 1]  -»  C[0, 1]  as 

(r*)(t)  -  x(t)  -  ta(l),  x  €  C[0, 1],  0  <  t  <  1 


Now  think  of  partitioning  our  original  output  process  Y  into  m  >  2  intervals  of  equal 
length  and  define  the  mapping  bm  :  C[0, 1]  — ►JR  by 

I  1/3 

(x(*/m)  -  x((i  -  l)/m)2  , 
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for  x  €  C(0, 1].  Finally,  the  g  function  of  interest  is  gm  *  6moI\  To  see  that  gm  corresponds 
to  the  batch  means  method  we  observe  that 


Jm(?n)  ■  m"1/J 


where 


fin 

Zi(n)  - 

Jd- 


in/m 
l)n/m 


Y(x)dx/(n/m) 


is  the  sth  batch  mean  of  the  process  {Y(i)  :  0  <  t  ^  n}.  Specializing  (S)  to  the  function 
gm  we  see  that 


mM 


as  n  — ►  oo,  where  tm_i  is  a  Student  Vt  random  variable  with  m  -  1  degrees  of  freedom. 
This  follows  from  the  fact  that  B(l)/gm(B)  is  distributed  as  tm-i  since  B  has  independent 
normal  increments.  For  other  examples  of  functions  g  €  M  for  which  the  distribution  of 
B(l)/g(B)  is  known  see  OLYNN  and  IGLEHART  (1989). 


4.  Variance  Reduction  Techniques. 

Once  a  basic  method  is  developed  to  produce  point  estimates  and  confidence  inter¬ 
vals  for  a  parameter  of  interest,  we  turn  our  attention  to  making  these  methods  more 
efficient.  Over  the  years  a  dozen  or  more  techniques  have  been  proposed  to  improve  sim¬ 
ulation  efficiency.  Good  references  for  many  of  these  techniques  are  BRATLEY,  FOX, 
and  SCHRAGE  (1987),  WILSON  (1984).  Here  we  have  elected  to  outline  three  of  these 
techniques. 

As  we  have  seen  in  Sections  2  and  3,  confidence  intervals  for  parameters  being  es¬ 
timated  are  generally  constructed  from  an  associated  CLT.  Each  CLT  has  an  intrinsic 
variance  constant,  say,  crj.  The  idea  for  many  variance  reduction  techniques  (VRT's)  is  to 
modify  the  original  estimate  in  such  a  way  as  to  yield  a  new  CLT  with  a  variance  constant 
<  <7a.  This  will,  of  course,  lead  to  confidence  intervals  of  shorter  length,  or  alterna¬ 
tively,  confidence  intervals  of  the  same  length  from  a  shorter  simulation  run.  Frequently 
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VKT's  are  based  on  some  analytic  knowledge  or  structural  properties  of  the  process  being 
simulated. 

The  first  VET  we  discuss  is  known  as  importance  sampling.  This  idea  was  first 
developed  in  conjunction  with  the  estimation  of  E{h(X)}  a  at ,  where  h  is  a  known  real¬ 
valued  function  and  X  a  random  variable  with  density,  say,  /.  Instead  of  sampling  X  from 
/,  we  sample  X  from  a  density  g  which  has  been  selected  to  be  large  in  the  regions  that 
are  “most  important”,  namely,  where  |/|  is  largest.  Then  we  estimate  at  by  the  sample 
mean  of  h(X)f(X)/g(X) ;  see  HAMMERSLEY  and  HANDSCOMB  (1964). 

This  same  basic  idea  can  be  carried  forward  to  the  estimation  of  parameters  associated 
with  stochastic  processes.  We  generate  the  process  with  a  new  probabilistic  structure  and 
estimate  a  modified  parameter  to  produce  an  estimate  of  the  original  quantity  of  interest. 
The  example  we  consider  here  is  the  M/Mf  1  queue  with  arrival  rate  A,  service  rate  /j, 
and  traffic  intensity  pm  \/p  <  1.  Let  V  denote  the  stationary  virtual  waiting  time  and 
consider  estimating  the  quantity  a  m  P{V  >  ti}  for  large  u.  When  p  is  less  than  one,  the 
virtual  waiting  time  process  has  a  negative  drift  and  an  impenetrable  barrier  at  zero.  Thus 
the  chance  of  the  process  getting  above  a  large  u  is  small,  and  a  long  simulation  would  be 
required  to  accurately  estimate  a.  The  idea  used  here  in  importance  sampling  is  to  generate 
a  so-called  conjugate  process  obtained  by  reversing  the  roles  of  A  and  p.  For  the  conjugate 
process  the  traffic  intensity  is  greater  than  one,  and  the  estimation  problem  becomes  much 
easier.  ASMUSSEN  (1988)  reports  efficiency  increases  on  the  order  of  a  factor  of  3  to  a 
factor  of  400  over  straight  regenerative  simulation  depending  on  the  values  of  p  and  u.  In 
general,  importance  sampling  can  yield  very  significant  variance  reductions.  Further  work 
along  these  lines  can  be  found  in  SIEGMUND  (1978),  GLYNN  and  IGLEHART  (1989), 
SHAHABUDDIN  et  al.  (1988),  and  WALRAND  (1987). 

The  second  VET  we  discuss  is  known  as  indirect  estimation.  Assume  we  are  interested 
in  estimating  a  a  E{X },  but  happen  to  know  that  £{Y}  »  aE{X }  4-  b  where  o  and  b  are 
known.  Sometimes  it  happens  that  a  CLT  associated  with  the  estimation  of  E{Y)  will  have 
a  smaller  variance  constant  associated  with  it  than  does  the  CLT  for  estimating  £’{A},  In 
this  case  we  would  prefer  to  estimate  E{Y }  and  we  use  the  affine  transformation  above  to 
yield  an  estimate  for  JS{A},  This  idea  has  proved  to  be  useful  in  queuing  simulations  where 
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the  affine  transformation  is  a  result  of  Little's  Law.  In  general,  variance  reductions  realized 
using  this  method  are  not  dramatic,  being  usually  less  than  a  factor  of  2.  For  further 
results  along  these  Unes,  see  LAW  (1975)  and  GLYNN  and  WHITT  (1986).  While  the 
affine  transformation  works  in  queuing  theory,  it  is  conceivable  that  other  transformations 
might  arise  in  different  contexts. 

The  third  and  final  VRT  we  discuss  here  is  known  as  discrete  time  conversion.  Suppose 
that  X  s  {X(t)  :  t  >  0}  is  an  irreducible,  positive  recurrent,  continuous  time  Markov 
chain  (CTMC).  Then  X(t)  =>  X  as  t  -+  oo  and  we  may  be  interested  in  estimating 
a  s  E{f(X)},  where  /  is  a  given  real-valued  function.  As  we  have  discussed  above,  the 
regenerative  method  can  be  used  to  estimate  a.  A  CTMC  has  two  sources  of  randomness: 
the  embedded  discrete  time  jump  chain  and  the  exponential  holding  times  in  the  successive 
states  visited.  The  discrete  time  conversion  method  eliminates  the  randomness  due  to  the 
holding  times  by  replacing  them  by  their  expected  values.  It  has  been  shown  that  this 
leads  to  a  variance  reduction  when  estimating  a.  Also,  as  an  added  side  benefit  computer 
time  is  saved  since  the  exponential  holding  times  no  longer  need  to  be  generated.  Gains  in 
efficiency  for  this  method  can  be  substantial.  Further  discussion  of  this  idea  can  be  found 
in  HORDIJK,  IGLEHART,  and  SCHASSBERGER  (1976),  and  FOX  and  GLYNN  (1986). 

5.  System  Optimization  Using  Simulation. 

Consider  a  family  of  stochastic  systems  indexed  by  a  parameter  9  (perhaps  vector- 
valued).  Suppose  a(0)  is  our  performance  criterion  for  system  9.  Our  concern  here  is  to  find 
that  system,  say  0o»  which  optimizes  the  value  of  a.  For  a  complex  system  it  is  frequently 
impossible  to  evaluate  or  analytically.  Simulation  may  be  the  most  attractive  alternative. 
We  could  naively  simulate  the  systems  at  a  sequence  of  parameter  settings  dj.dj,  •  •  ■ ,  0* 
and  select  setting  that  optimizes  a(9i).  In  general  this  would  not  be  very  efficient,  since 
k  would  have  to  be  quite  large.  A  better  way  would  be  to  estimate  the  gradient  of  a  and 
use  this  estimate  to  establish  a  search  direction.  Then  stochastic  approximation  and  ideas 
from  non-linear  programming  could  be  used  to  optimize  a. 

T  x>  general  methods  have  been  proposed  to  estimate  gradients:  the  likelihood  ratio 
method  and  the  infinitesimal  perturbation  method.  We  will  discuss  both  methods  briefly 
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Suppose  X  a  {J£„  :  n  >  0}  is  a  discrete  time  Markov  chain  (DTMC)  and  that  the  cost  of 
running  system  9  for  n  + 1  steps  is  g(9,  Xo, . . . ,  Xn).  The  expected  cost  of  running  system 
9  is  then  given  by 

a(9)S*E,{g(6,X  . . *„)},  (6) 

where  Et  is  expectation  relative  to  the  probability  measure  P{9)  associated  with  system  9. 
If  Et{‘)  were  independent  of  0,  we  would  simply  simulate  iid  replicates  of 
Vg(9,Xo,. .  .,Xn).  By  introducing  the  likelihood  function  L(d,  A'o, . . .  ,Xn)  it  is  possi¬ 
ble  to  write  a(9)  as 


«(*)  -  B*  {*(*,  X0}  •  •  • ,  Xn)£(0,  X;  —  ,  Xn)} 

for  a  fixed  value  of  0O.  Then  we  can  write 

Va(9)  -  E,0  {Vg(6y  X;  •  •  • ,  *»)L(0,  JTo,  •  ■ •  • ,  *„)}, 

where  the  interchange  of  V  and  must  be  justified.  A  similar  approach  can  be  developed 
to  estimate  the  gradient  of  a  performance  criterion  for  a  steady-state  simulation.  For  an 
overview  of  this  approach  see  GLYNN  (1987),  and  REIMAN  and  WEISS  (1986). 

The  second  method  which  has  been  proposed  for  estimating  gradients  is  called  the 
infinitesimal  perturbation  analysis  (IPA)  method.  In  this  method  a  derivative,  with  respect 
to  an  input  parameter,  of  a  simulation  sample  path  is  computed.  For  example,  we  might 
be  interested  in  estimating  the  mean  stationary  waiting  time  for  a  queueing  system  as  well 
as  ita  derivative  with  respect  to  the  mean  service  time.  Since  we  are  taking  a  derivative 
of  the  sample  path  inside  an  expectation  operator,  the  interchange  of  expectation  and 
differentiation  must  be  justified  in  order  to  produce  an  estimate  for  the  gradient  V«(0), 
say.  The  IPA  method  assumes  that  if  the  change  in  the  input  parameter,  0,  is  small 
enough,  then  the  times  at  which  events  occur  get  shifted  slightly,  but  their  order  does 
not  change.  It  has  been  shown  that  the  IPA  method  yields  strongly  consistent  estimates 
for  the  performance  gradient  in  a  variety  of  queueing  contexts;  see  HEIDELBERG ER, 
CAO,  ZAZANIS,  and  SURI  (1988)  for  details  on  the  IPA  method  and  a  listing  of  queueing 
problems  for  which  the  technique  works. 
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Bayesian  Inference  for 
Wsibull  Quantiles 
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The  posterior  distribution  of  a  two  parameter  Weibull 
quantile  for  a  noninformative  prior  may  be  obtained  exactly 
(Bogdanoff  and  Pieros,  1973),  although  the  necessary 
numerical  integration  detracts  from  the  usefulness  of  this 
result.  Credible  intervals  for  this  posterior  have  an 
alternative  frequentist  interpretation  in  terms  of 
conditional  tolerance  limits  (Lawless,  1975). 

An  approximation  to  the  Lawless  procedure  was  proposed  by 
DiCiccio  (1987).  This  approximation  does  not  involve 
numerical  integration  and  is  of  order  0  (n  '  )j  apparently 
it  is  adequate  even  for  samples  as  small  as  ten. 

The  focus  of  this  paper  is  on  the  use  of  DiCiccio' s 
result  for  the  routine  calculation  of  Weibull  quantile 
posteriors.  Even  a  nonbayesian  may  find  the  posterior  cdf's 
useful  since  they  provide  an  easy  graphical  means  for 
obtaining  accurate  tolerance  limits. 

Examples  from  strength  data  for  composite  materials  are 
presented  and  a  specific  application  of  importance  to 
aircraft  design  is  discussed. 
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The  Valbull  nodal  is  widely  used  to  represent  failure  data  In  engin- 
aarlng  applications.  On*  raason  la  baeausa  tha  Valbull  distribution  is  tha 
Uniting  distribution  of  tha  suitably  noraalizsd  minimum  of  a  sample  of 
positlva  lid  random  varlablas  undar  qulta  ganaral  conditions  (Barlow  and 
Proschan,  1975,  ch.  6).  Tha  nodal  is  tharafora  approprlata  for  tha  strength 
of  a  systan  conposad  of  a  string  of  nany  links  whara  tha  strangth  of  tha 
links  ara  lid  and  tha  systan  fails  whan  tha  waakast  link  falls  (Bury,  1975, 
ch.  16).  An  axanple  of  a  physical  systan  which  can  ba  nodalad  in  this  way 
is  tha  strangth  of  a  brittle  fiber  in  tension,  Another  raason  why  tha 
Valbull  nodal  is  used  is  that  tha  distribution  is  vary  flexible  and  con¬ 
sequently  it  often  fits  data  wall. 

Inference  for  tha  Valbull  distribution  (or,  equivalently,  for  tha 
extrema  value  distribution,  which  is  tha  distribution  of  tha  logarithm  of  a 
Valbull  random  variable)  is  complicated  by  tha  fact  that  tha  Valbull  is  not 
in  tha  exponential  family,  and  consequently  tha  minimal  sufficient  sta¬ 
tistics  ara  tha  entire  sample.  Also,  although  MLE's  ara  easily  obtained 
iteratively,  tha  distributions  of  tha  MLE’s  and  plvotals  based  on  tha  MLE’s 
can  not  ba  obtained  in  closed  form.  Tha  same  is  true  of  linear  estimators 
of  tha  Valbull  parameters. 

At  least  three  approaches  to  Valbull  inference  have  been  taken.  The 
first  is  to  tabulate  approximate  quantiles  of  the  plvotals  obtained  by  Monte 
Carlo.  From  these  tables  one  can  obtain  confidanca  intervals  on  parameters 
as  wall  as  confidence  intervals  on  quantiles  (tolerance  limits)  for  complete 
samples  (Thoman,  Bain  and  Antla  1969,  1970).  A  problem  arises  for  incom¬ 
plete  samples,  since  tables  must  be  prepared  by  simulation  for  each  censor¬ 
ing  configuration.  The  tables  which  have  been  prepared  (Bi liman,  Anti*  and 
Bain,  1972)  are  Inadequate  A  second  appraoch  is  to  approximate  the  dis¬ 
tribution  of  the  plvotals  (e.g.  Lawless  and  Mann,  1976).  Thasa  approxima¬ 
tions  are  empirical  and  consequently  they  are  not  very  satisfactory  from 
theoretical  point  of  view. 
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Finally  we  reach  Che  third  approach,  which  it  tha  focus  of  this  paper. 
For  any  location-scale  faaily  (a.g.  tha  extrene  value  faaily)  and  any  equiv- 
ariant  estiaators  of  the  paraaeters  (e.g.  MLE's)  tha  distribution  of  certain 
pivotals  can  by  obtaioned  exactly  if  one  conditions  on  the  ancillary  sta¬ 
tistics.  From  these  pivotals  one  can  get  exact  conditional  eonfidence 
bounds  and  toleranoe  Units  for  any  saaple  slse.  The  nethod  Is  applicable 
to  both  complete  and  Type  II  censored  saaples  (i.e.,  samples  for  which  only 
the  r  smallest  order  statistics  are  observed)  and  requires  no  tables. 

Since  the  Intervals  have  exact  conditional  confidence,  it  follows  that  they 
are  also  exact  unconditionally.  In  addition,  this  nethod  has  the  advantage 
of  making  use  of  all  of  the  information  with  respect  to  the  parameters  which 
is  in  the  data  (the  parameter  estimates  are  in  general  not  sufficient 
statistics),  though  for  the  Walbull  model  this  does  not  appear  to  be  a 
practical  concern  (Lawless,  1973).  This  conditional  approach  is  apparently 
due  to  Lawless,  who  introduced  it  in  (Lawless  1972).  An  exposition  of  the 
procedure  appears  in  (Lawless,  19S2),  which  is  also  useful  as  a  guide  to  the 
literature. 

If  one  chosea  an  appropriate  noninformative  prior  distribution  for  the 
parameters  of  a  location-scale  family,  then  the  posterior  distribution 
either  of  the  parameters  or  of  a  quantile  conditional  on  the  ancillarles  are 
formally  identical  to  frequentlst  confidence  and  one-sided  tolerance  limits 
respectively. 

Bayesian  and  frequentlst  terminology  may  thus  be  interchanged  freely 
and  I  will  do  so  in  this  paper.  This  is  particularly  valuable  when 
discussing  tolerance  limits,  which  have  a  frequentlst  Interpretation  which 
la  difficult  for  nonstatistician?  to  understand.  A  posterior  cdf  of  a 
quantile,  however,  is  immediately  understood  intuitively.  Such  a  cdf  can  bv 
used  to  obtain  graphically  arbitrary  one  sided  and  approximate  two  sided 
conditional  tolerance  limits  since  for  tha  cases  discussed  herein  these 
intervals  coincide  with  noninformative  prior  Bayesian  credible  Intervals. 

The  main  disadvantage  of  this  conditional  approach  is  that  it  is  com¬ 
putationally  intensive.  Many  numerical  integrations  must  be  performed  for 
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•ach  dataset  as  one  iteratively  approximates  the  confidence  limit. 

One  goal  of  this  project  has  been  to  implement  the  Lawless  procedure 
for  the  extrame  value  distribution  in  a  *  robust*  FORTRAN  program  which  can 
be  used  with  little  user  interaction.  Another  goal  has  been  to  investigate 
a  recent  approximation  to  the  conditional  procedure  (DICiccio,  1987)  which 
Is  accurate  to  0p(n’,/a).  This  approximation  makes  the  the  calculation  of 
posterior  distributions  feasible,  A  FORTRAN  program  to  calculate  and  plot 
Che  posterior  distribution  of  Velbull  quantiles  which  makes  use  of  the 
DICiccio  result  is  discussed.  The  results  of  a  small  simulation  to  assess 
the  accuracy  of  the  approximation  are  presented,  though  little  effort  was 
spent  on  the  simulation  since  the  order  of  convergence  in  probability  has 
been  determined. 

2.  Zhi  HftlbuU  P.U&ilbutlfln 

The  cdf  of  the  Woibull  distribution  is 
F(x;  a,  g)  -  1  -e“^x  'W* 

where  0  is  a  scale  parameter  and  a  a  shape  parameter.  Maximum  likelihood 
estimation  is  straightforward,  The  following  equation  is  solved  by  Newton- 
Raphson  for  a: 


<X‘  x/  log  xA)  (I*  xA“)‘l  -l/o  -  1/r  I  log 

where  £  x4  &  . . .  S  xr  are  the  order  statistics,  n  i  r  is  the  sample 

size,  and 


-  I  "j.  +<n  -r)  wr  . 

i-i 


A  FORTRAN  subroutine  'WEIMLE'  for  performing  these  calculations  is  given  in 
tha  appendix. 
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2.  Xhft  Extreme  Value  DUtributiop 

Let  X  be  distributed  Wei bull  with  shape  «  end  seele  0,  The  dlstrlbu- 
tion  of 


V  -  log(X) 


la 


H(y;  u,  b)  -  G((y*u>/b)  -  exp  (-exp  <(y*u)  /b)> 


where 


b  -  l/o  and  u  -  log  fi 

are  scale  and  location  parameters  respectively.  The  location-scale  family 
H(y;  u,  b)  is  called  the  extreme  value  distribution.  Results  for  the 
extreme  value  distribution  are  easily  interpreted  in  terms  of  the  Veibull 
distribution,  and  vice  versa. 


k-  Conditional  Inference  lax  location- scale  Faailisi 

The  presentation  below  follows  Lawless  (1982).  The  distribution  H(y; 
u,  b)  is  taken  to  be  the  extreme  value  distribution  as  in  the  previous 

A  * 

section.  The  parameter  estimates  u  and  b  may  be  taken  to  be  MLE's,  but  the 
results  hold  for  any  equivariant  estimators  *  -  that  is  any  statistics  u  and 
E  which  satisfy 


(Kdyj+c,  ....  dyt+c)  -  d  u(y1 . yr)  +  c 

E(dyj+c . dyt+c)  -  d  B(yj . yr) 

for  any  c  and  any  d  >  0.  The  maximum  likelihood  estimates  are  reodily  seen 
to  be  equivariant. 
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Lat  the  aanpla  alza  ba  n  and,  to  allow  for  Typa  II  cansoring,  lat  r  £  n 
ba  tha  nunbar  of  data  valuaa.  Danota  tha  dam 1 ty  of  G(*)  by 

0'<(y-u)  /b)  -  1/b  g(<y-u)  /b) 

flrat  «a  danonatrata  that  tha  following  randon  varlablaa  ara  pivotal;  that 
la  thay  hava  probability  dlatrlbutlona  which  do  not  dapond  on  tha 

paraaatara ; 

A  A  *  A 

Zx  -  (u  -u)  /b  Za  •  b  /b  Z,  -  (u  -u)  /b. 

Lat  yi  i  S  r  ba  tha  ordar  atatiatica  of  a  random  aanpla  from  H(y) .  Conaidar 
tha  randon  varlablaa 

-  <yt  ~u>  /b  . 

Tha  wt  ara  tha  ordar  atatiatica  of  a  randon  aanpla  fron  G<‘)  and  hanca  ara 

A 

obvioualy  pivotal.  Sinca  tha  aatinator  u  la  aaaunad  to  ba  aqulvariant  wa 
hava  that 

u(w: . vt)  -  u((yA-u)/b . (y^uj/b)  - 

1/b  (u(ylt  ....  yt)  *u>  -  (u-u)/b  -  Z, . 

Hanca  Z3  la  pivotal.  Similarly,  Z2  la  a  pivot  alnca 

A  A 

b(w1 . Vj.)  -  b(<y1-u)/b,  ...,  (yr-u)/b)  - 

b<ya . yt>  /b  -  Z2 

Finally, 

Zj  -  (G-u)/b  -  <(G-u)/b)  (b/b)  -  z3/z2 

A  A 

Tha  quantitiaa  at  -  (x^uj/b  ara  Innadlataly  aaan  to  ba  ancillary  since 
tha  (a^  ara  a  random  aanpla  fron  G('),  whara  G(’)  la  a  complataly  known 
diatrlbution.  Only  r-2  of  thaaa  ancillary  atatiatica  ara  indapandant  since 
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tt<«l , 


•  •  ■  » 


(u<yx,  ....  yg )  -u)/b  -  0 


and 

0 

0 

A  A  A 

•, )  ■  b(yx  •  •  •  •  •  y*  )A  ■  i  • 

The  fundamental  result  upon  which  conditional  inference  from  e  frequen- 
tlat  perspective  le  beied  le  that  the  Joint  pdf  of  (Z,  ,  Z2 ,  ax ,  . a(.2) 
la  of  the  form 


hCtj,  *a.  fi)  -  k(a,  r,  n)  s2*‘l  (  ft  |(a1  +  *l*2))  [  G(ae*a  +  *»*«> ]**'* 

!■  1 

where  k(a,  r,  n)  la  a  function  of  a^  aJt  ....  ar|  only.  The  pdf  of  {Zx, 

: 2 )  given  a  la  of  the  came  form  aa  h  above  except  that  the  normalizing  con¬ 
stant  ia  different. 

The  proof  la  atraightforward.  Begin  with  the  joint  pdf  of  (y^,  . 
yr)  and  make  the  change  of  variables 


y4  -  b  at  +  G. 

Hie  Jacobian  of  this  transformation  is  a  constant  given  a.  A  second  change 
of  variables 


A 

u 


Zx  b  +  u  -  bZ:Z2  +  u 


b  -  Z2b 


gives  the  desired  result. 
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k-  Confidence  Intervals  £si  Z&ZXWt  Yalue  Quant lift! 

Using  the  pivotal  density  derived  in  the  previoue  section,  it  is  not 
difficult  to  obtain  exact  confidence  intervela  on  quantiles  of  the  extreme 
value  (or  equivalently,  the  Veibull)  distribution.  Toward  this  end,  ve 
determine  the  distribution  for  the  scale  paraaeter  pivotal  Zt.  This  result 
la  of  Interest  In  it's  own  right  since  it  leads  to  confidence  intervals  on 
the  extreme  value  ecale  (or  Veibull  shape)  paraaeter.  To  get  the  density  of 
Z2 ,  merely  Integrate  out  Z1  from  the  joint  pdf  given  in  the  previous 
section,  giving 

h(«a  | a)  -  k(a)  exp  (la^j)  *ar'1  /tl*  sxp  (a^)]*. 

Next,  make  the  change  of  variables 

Zp  -  Zt  -Wp/Za  and  Za  -  Za 

where 


wp  -  In  (-In  (1-p)). 

The  Joint  density  of  Za  and  Zp  is 

zal  4)  *  V*  *XP  (I  +  V*  +  V 

•  exp  (-1*  exp  (a^j  +  zpza  +  wp)). 

Tho  cdf  of  Zp  can  bo  *xpr*jaed  In  torna  of  tho  danaity  of  Zp  and  Z2  as 


•  t 

P(Zp  St)-  /dza  X  dip  f(Zp,  Zs I  a). 

0  0 

Change  variables  again,  this  time  letting 

y  -  exp  (ZpZj  +  wp)  I*  exp  (*tz2)  and  z?  -  za . 
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The  double  integral  can  now  be  written  aa  a  ■ ingle  integral  by  recognizing 
that  the  integral  over  y  la  the  incomplete  gamma  function: 


P(Zp  S  t)  -  I  da,  h(za  | a)  I (exp  (t*a  +  wp)  I*exp  (a^),  r) 
0 


where 


1<«,  r)  -  (l/r(r)>  !  x*’1  e’“  dx. 
o 

Since  the  pth  quantile  of  an  extreme  value  random  variable  la 

*p  -  u  +  V*' 

the  pivotal  Zp  can  be  expressed  as 

Zp  -  <u  •  xp)  /b. 

The  probability  distribution  of  Zp  can  therefore  be  used  to  obtain 
exact  conditional  confidence  Intervals  on  extreme  value  quantiles.  One 
first  obtains  the  constant  of  integration  k(a)  numerically.  Next,  the 
P(Zp  S  t)  is  evaluated  numerically  for  several  choices  of  t  until  the 
quantile  of  the  distribution  of  Zp  is  determined  to  the  desired  accuracy. 
Finally,  the  confidence  bound  on  xp  is  trivially  obtained  by  pivoting. 


6.  A  fliailiB  Interpretation 

Independently  of  Lawless,  Bogdanoff  and  Pierce  (1973)  arrived  at 
results  identical  fo  those  outlined  above  from  a  Bayesian  point  of  view. 
Bayesian  results  are  much  easier  to  explain  to  nonstatisticians .  This  la 
particularly  true  for  the  problem  that  I'm  prlnarily  Interested  in,  con¬ 
fidence  intervals  on  a  quantile,  and  the  advantages  of  Bayesian  motivation 
for  a  particular  application  will  be  discussed  in  a  later  section. 
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Let  { y ^ )  be  the  order  statistics  of  a  Type  II  censored  sample  of  size  r  $  n 
from  an  extrema  value  dletrlbution.  The  uaual  Joint  noninformativa  prior 
for  the  location  parameter  (u)  and  scale  paraaeter  (b)  of  a  location* scale 
faaily  is: 

#« 

*(u)  *  constant  «(log(b))  *  constant 
«(u,  b)  «  1/b. 

Using  the  expression  for  the  extreae  value  pdf  given  in  e  previous 
section,  the  corresponding  posterior  distribution  is  seen  to  be 

K 

*<u,b  |y)  «  b  exp  (I  (yt  •  u)/b) 

i-l 

,xp  <•!*  exp  ((y^  *u)/b)) , 

The  location  paraaeter  is  readily  Integrated  out  giving 


*(b  |y)  «  (l/b)  *  exp  (I  yt/b>  /[l*  exp  (yt  /b) ]  r  , 

i-l 

The  normalizing  constant  is  determined  by  numerical  integration.  Inference 
based  on  this  result  will  be  shown  in  the  next  section  to  be  formally  equiv¬ 
alent  to  Welbull  Inference  using  the  pivotal  for  the  shape  parameter. 

Let  ♦(u,  b)  be  any  scalar  function  of  the  parameters  about  which  in¬ 
ference  la  to  be  made.  Assume  that  VKu,  ')  is  monotonically  increasing  in  u 
for  fixed  b,  If  a  function  can  bo  found  which  satisfies  this  condition 
piecewise,  then  the  following  results  may  still  be  applied  to  each  monoconic 
section  of  the  function.  Some  useful  choices  for  ♦  are 

*(u,  b)  -  u  (location  parameter) 

♦(u,  b)  -  u  +  log(-log  (1-p))  b  (pth  quantile) 
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♦(u,  b)  -  exp(-exp((t-u)/b)))  (reliability  at  tine  t) 

♦(u,  b)  -  u  •  yb  (naan;  y  ia  Euler' a  eonatant) 

Define  the  lnverae  function  17(8,  b)  by  means  of  the  relation 

V*[f)(a,  b),  b]  -  a. 

The  posterior  cdf  of  \j>  can  be  expressed  as 

m 

P  (VK u,  b)  S  1  |y)  •  /  P(u  S  rj(a,  b)  |b,  y)  *(b  |y)  db 

0 

It  is  easy  to  show  that  the  conditional  distribution  of  \  -  exp  (-u/b)  given 
b  ia  the  following  gamma  distribution: 

*(*•  |b,  y)  -  (l/r(r))  [ X* exp  (yt/b))  *r'x  «xp  <•  h  I*«xp  (yLA>)). 

Simple  algebra  also  shows  that 

P  (u  S  r|(a,  b)  | b ,  y)  -  P(\  i  exp(*r)(o.  b)/b)  |b,  y) . 

Combining  these  results,  we  have  finally  that 

P  (♦(u.b)  S  a  |y)  - 

/(I  *I(exp  (•n(s,b)/b)  X*exp  (yt/b),  r)>  *(b  |y)  db 
o 

where  1(6,  r)  denotes  the  Incomplete  gamma  function 

9 

1(6, r)  -  (l/r(r))  /  xr_1  exp  (-x)  dx. 

0 
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For  a  confidence  bound  on  the  pth  quantile  x  , 


»j(e,b)  -  #  *  vpb 


P<xp  <•)«/(  1  -  I  (exp  (wp  •«/ b  I*  exp  (y^),  r)  *(b  |y)  db. 
0 


The  feet  that  for  Inference  about  quantllee  and  about  the  shape  (or,  in 
terns  of  the  extreme  value  distribution,  scale)  parameter  the  Bayesian 
approach  is  equivalent  to  the  Lawless  conditional  approach  will  be  demonstr¬ 
ated  next. 


2.  formal  EauiYilmcf  si  Btyialin  And  En.aatn.tUt  Results 

First  we  demonstrate  that  posterior  intervals  for  the  scale  parameter 
(b)  have  an  exact  frequency  interpretation.  Let  b^  be  *uch  that  the  post¬ 
erior  probability  that  b  is  greater  than  b^  is  y.  Since 

P(b  js  b:)  -  P(b/b  s  b/bj)  -  y, 

we  make  the  change  of  variable 

A 

i  -  b/b 

and  substitute  for  the  y1  in  terns  of  the  a1  to  get 


y  -  /*(b  |y)  db  -  ;  exp  (Iy4 /b)  b*r  /(I*exp  (y4 /b) ]'  db 


«  /  exp  (Za1z)/[Z*exp  (a^)]'  r*'2  dz 
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'  l 

•  X  ex P  (Xa1B)/[Z*«ap  (*!*)]*  zr*a  dz 
e 


A 

b/bx 

-  X  h(z  |a)  dz. 
o 

To  see  th*  equivalence  of  the  reaulta  for  a  quantile,  wo  make  tho  sub- 
■titueion 


t 


A 


A 


(u  -  z)/b 


and  noto  that 

oxp  <vp  -  a/b  < I* exp  - 

oxp  (wp  +  tb/b  ■  u/b  (X*exp  (a^/b  +  u/b)))  - 
oxp  (wp  +  tb/b  <X*oxp  (aib/b))) . 

A 

Tho  chango  of  varlablo  z  -  b/b  glvoa  tho  doslrod  rooult. 


&■  ZbA  L&e  giLpftialUfii  fiaflOii  Pjjt,t,ilbu.filan 

Tho  probability  distribution  of  a  generalized  gamma  random  varlablo  T 

la 

Ft(x;  a,  fi,  k)  -  <«/r<k))  (x*k'l/^k)  «xp  (-(x/jD"). 

Dotalla  on  inference  for  thia  family  may  be  found  In  (Farowoll  and  Prentice, 
1577)  and  (Lavloaa,  1980),  Note  that  tho  case  k  ■  1  corresponda  to  the 
Velbull  dlatribution. 
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If  T  has  a  generalized  gamma  distribution,  than  Y  -  log  (T)  can  be 
written  in  the  fora  p  -  etf  where 

M  -  log<0)  +  log(k )/at 

a  -  l/(«kl/1) 

and  V  haa  the  probability  density 

fH(w;  k)  »  k*‘1/a  /r(k)  exp  (kl/Jw  *  exp  (w/k1/a)). 

Y  is  said  to  have  a  log  generalized  gamma  distribution. 

By  varying  k,  one  obtains  a  family  of  location- scale  dsitrlbutions 
ranging  from  the  normal  (k  *  •)  to  Che  extrema  value  <k  -  1) .  Although  we 
will  restrict  attention  to  the  case  k  -  1,  it  is  straightforward  to  adapt 
both  the  frequentist  and  the  Bayesian  approaches  to  arbitrary  fixed  k  and 
even  to  certain  regression  situations  (Jones  et.  al.,  1980). 

£.  Approxlniti  Infartno  iar  iht  Log  6«n«rillgifl  £iauu  Distribution 

Let  J7  and  a  denote  maximum  likelihood  estimates  of  p  and  a  subject  to 
the  constraint 

-  ? 4  v* 

that  la,  the  HLE  of  the  pth  quantile  is  required  to  equal  ypQ.  If  p  and 
denote  the  unconstrained  MLE's,  and  if  L(p,  o)  denotes  the  log  of  the  log 
generalized  gamma  likelihood,  then  the  asymptotic  distribution  of  the  sta¬ 
tistic 


V(yp0)  -  *2[L(p,  5)  -  L(p,  a)) 

Is  x3  with  one  degree  of  freodom.  Lawless  (198i,  sec.  4.2)  suggests  that 
Inference  based  on  V(yp[J)  is  acceptable  for  moderate  to  large  samples  but 
that  the  approximation  may  be  Inadequate  for  small  samples. 
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Q  > 


DiCiccio  (1987)  has  applied  general  technique*  of  Barndorf •Nielson 
(1986)  in  order  to  develop  a  computationally  inexpensive  modification  to  the 
signed  square  root  of  V(yp0)  which  yields  a  likelihood  ratio  based  approxim* 
atlon  suitable  even  for  quite  small  sample  sizes.  The  numerical  Integration 
required  by  the  exact  methods  Is  only  troublesome  for  moderate  to  large 
samples;  so  the  approximation  is  actually  of  questionable  use  over  the  range 
of  sample  sizes  for  which  it  is  inaccurate. 

I  will  not  reproduce  the  details  of  the  DiCiccio  approximation  here  for 
two  reasons.  The  most  important  of  thase  la  that  only  the  results  are 
presented  in  (DiCiccio  1987)  and  to  repeat  these  results  without  having 
studied  their  derivation  would  serve  no  purpose.  A  second  reason  is  that 
although  the  approximation  is  inexpensive  to  compute,  the  formulas  are 
messy,  and  to  reproduce  thsm  here  is  to  invite  typographical  errors.  Inter¬ 
ested  readers  should  refer  to  (DiCiccio  1987)  and  to  the  FORTRAN  Implementa¬ 
tion  as  subroutine  LAWAPX  in  the  appendix. 


lfi.  Tht  Ac.gmgfix  si  ApRig.Kiaifcl.aa 

The  DiCiccio  approximation  can  be  shown  to  be  accurate  to  Op(n'3/2) 
(DiCiccio,  1987,  p,  37),  so  an  extensive  simulation  study  of  accuracy  is  un¬ 
necessary.  The  results  of  a  very  small  such  study  are  presented  in  Table  1. 
Sampltf  of  sizes  ranging  from  10  to  30  were  taken  from  Walbull  populations 
with  different  shape  parameters.  Both  the  Lawless  and  the  DiCiccio  methods 
were  used  to  calculate  95  percent  lower  confidence  limits  on  the  tenth  per¬ 
centiles  of  the  Walbull  populations  and  the  mean  and  standard  deviation  of 
the  percent  different  between  the  Lawless  result  and  the  DiCiccio  approxima¬ 
tion  were  calculated  for  100  replicates  for  each  case.  One  would  expect 
that  the  approximation  error  should  be  a  rapidly  decreasing  function  of  n, 
and  this  Is  observed  to  be  the  case.  The  quality  of  the  approximation  is 
also  seen  to  be  a  function  of  the  shape  parameter  of  the  population. 

Halving  the  shape  parameter  (from  10  to  5)  approximatly  doubles  the  mean 
percentage  error  uniformly  over  sample  sizes.  Also,  the  approximation  error 
actually  appears  to  be  a  function  of  the  number  of  uncensored  values  rather 
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than  the  overall  sample  size,  which  is  not  surprising.  Overall,  the 
DiCicclo  result  appears  to  be  satisfactory  for  samples  of  10  or  more  un¬ 
censored  values,  and  remarkably  good  for  samples  of  30  or  mors  observed 
values.  This  conclusion  is  based  partly  on  the  small  simulation  presented 
here  and  partly  on  experimenting  with  various  cases  of  real  and  simulated 
data, 


11.  Ad  ABgU&illflD:  fiaaaailli  ttitariil  lull  Values 

A  criterion  used  both  by  aircraft  designers  whan  choosing  a  material 
for  a  specific  application  and  by  the  Federal  Aviation  Adainiatrstion  when 
certifying  a  new  material  for  s  structural  aircraft  application  is  the 
material  basis  value.  A  'B-basis  value*  is  defined  to  be  a  lower  95  percent 
confidence  limit  on  the  tenth  percentile  of  the  strength  distribution  of  a 
material  and  an  'A-basis  value*  is  a  95  percent  lower  confidence  limit  on 
the  first  percentile,  The  reason  for  thase  tolerance  limits,  which  have 
bean  used  in  the  industry  for  decades,  is  that  a  designer  is  primarily 
Interested  in  the  lower  tall  of  the  strength  distribution.  In  order  to 
design  a  reliable  structure,  he  would  like  to  estimate  the  stress  level  at 
which  a  material  is  90  percent  or  99  reliable.  A  tolerance  limit  is  an 
attempt  to  estimate  these  quantiles  in  a  conservative  way.  Such  conserva¬ 
tism  is  particularly  necessary  for  advanced  composite  materials,  which  typi¬ 
cally  have  relatively  high  strength  variability.  Also,  advanced  materials 
are  generally  expansive  to  manufacture  and  test,  resulting  in  small  sample 
sizes . 

The  work  presented  hare  has  bean  motivated  by  a  need  for  Improved 
methodology  for  calculating  basis  values  and  for  communicating  lower  tall 
quantile  information  to  the  engineer.  Typically,  the  engineer  who  routinely 
calculates  and  Interprets  these  numbers  has  little  appreciation  for  the 
rather  convoluted  frequency  arguments  behind  tolerance  limits.  The  long  run 
proportion  of  times  a  statistic  calculated  from  successive  samples  of  size  n 
from  a  hypothetical  population  is  greater  than  a  certain  quantile  of  that 
population  is  of  little  help  to  the  statistically  naive.  The  simple  state¬ 
ment  that  the  tenth  (first)  percentile  is  greater  than  the  B-basls  (A-basis) 
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value  with  95  percent  probability  ia  ouch  more  direct  and  intuitive.  Also, 
the  Bayesian  approach  presents  all  of  the  Information  in  the  data  about  the 
lover  tall  quantile  of  interest,  which  is  what  should  be  the  ultimate 

concern  of  the  engineer  anyway,  The  fact  that  the  tolerance  limit  is  only  a 

convenient  summary  statistic  of  this  distribution  becomes  clear  when  the 
user  is  presented  with  the  entire  posterior  and  shown  hov  to  determine 
arbitrary  tolerance  limits  graphically. 

Table  2  presents  B-basis  value  calculations  for  a  graphite  flber/epoxy 
material  made  by  four  fabricators.  Note  the  agreement  between  the  DlCiccio 
and  the  Lawless  calculations.  Figure  1  consists  of  the  four  tenth  per* 

candle  posteriors.  Not  only  do  two  of  the  fabricators  have  nearly  the  same 

B-basis  value,  they  also  have  virtually  identical  quantile  posteriors. 
Several  questions  Immediately  come  to  mind:  Vhy  did  the  other  two  manufac¬ 
turers  produce  substantially  weaker  material?;  Are  other  lower  tall  quantile 
posteriors  for  the  two  'similar*  fabricators  as  close  together?;  etc.  Ex¬ 
amining  the  posterior  rather  than  a  summary  statistic  of  the  posterior  leads 
to  insight  into  the  data  that  might  not  otherwise  be  apparent.  Figure  2 
demonstrates  that  the  B-basis  value  can  be  retrieved  graphically. 

Table  3,  Figure  3  and  Figure  4  present  corresponding  results  for 
another  material:  woven  Kevlar  fibers  in  an  epoxy  matrix.  These  data  show 
much  less  fabricator -to-fabricator  variability  than  do  the  graphite/epoxy 
data.  This  can  readily  be  seen  from  the  tolerance  limit  calculations.  The 
fact  that  there  is  essentially  no  evidence  in  the  data  to  suggest  that  the 
fabricators  differ  with  respect  to  the  tenth  perce  idles  of  their  strength 
distributions  is  made  particularly  clear  by  the  'overlapping'  posteriors  of 
this  quantile. 


11-  Cgncluilac 

This  paper  reviews  two  results  related  to  conditional  Inference  in  lo¬ 
cation-scale  families,  emphasizing  inference  on  Velbull  quantiles.  These 
methods  are  due  to  lawless  (1972)  and  Bogdanoff  and  Pierce  (1973).  For  the 
case  of  inference  on  quantiles  both  procedures  are  equivalent,  though  the 
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former  la  motivated  by  frequency  considerations,  whlla  tha  lattar  ia  derived 
from  a  Bayaalan  point  of  view.  Tha  raeant  work  of  DiClcclo  (1987)  greately 
reduces  tha  computational  bur dan  of  both  mathoda  with  littla  loaa  of 
accuracy. 

Tha  advantages  of  tha  Bayaalan  intarpratationi  at  laaat  for  lnfaranca 
an  quantilaa,  haa  baan  demonstrated  by  maana  of  an  axampla  from  an  engin* 
aaring  application. 
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TABLE .1 


AMucicy  laaifflUattlan 

A  slaulstlon  of  95 1  lower  confidence  bound*  on  t he  10th  psreantlls  using  Che 
I Veibull  distribution  with  100  rspllestss  per  ess*  wee  perforated.  The 
rssults  ere  sussasrlssd  below: 


n 

r 

Shape 

Scale 

At 

Stan,  error  of 

10 

10 

10 

1 

1.09 

.029 

20 

20 

10 

1 

.380 

.0086 

10 

10 

5 

1 

2.20 

.039 

20 

20 

3 

1 

.761 

.017 

30 

20 

3 

1 

.732 

.016 
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TABLE  2 

Carbon  fiber  /  Epoxv  specimen  tensile  strength  data 

9.S.I.LC8  qp  lQsh  jttiaiatilt 


Fabricator 

n 

Estimates 

(KSI) 

Lawless 

DiCiccio 

A 

48 

244.1 

244.3 

B 

36 

271.4 

271.6 

C 

33 

228.2 

228.3 

D 

23 

269.5 

269.8 

TABLE  3 

Ktttlai  fibril  IPQXV  ■aecinan  tensile  strength  data 

111,  1C B.  an  ,.lQth  pirctntllt 


Fabricator 

n 

Estimates 

(KSI) 

Lawless 

DiCiccio 

A 

23 

77.38 

77.64 

B 

18 

76.36 

76.30 

C 

30 

77.18 

77.23 

D 

10 

78.45 

78.64 
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Figure  1 
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AS-4/3901-6  RTD  Tension 


Kevlar/Cycom  RTD  Tension 
10th  Percentile  Distributions 


Kevlar/Cycom  RTD  Tension 


Stress  (KSI) 


Appendix:  FORTRAN  Listings 


The  following  programs  were  developed  on  an  Alliant  FX/8  and  should  run  with 
little  modification  on  any  32  bit  machine.  However ,  the  eoftware  has  not  been 
tested  to  the  point  where  it  can  be  considered  error  free.  The  programs  are 
provides  as  a  guide  to  an  individual  wishing  to  implement  the  algorithms  discussed 
in  this  paper. 
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progra*  laupg* 

Mart  Vangat*  5/?0/88 

Progra*  to  1*pl**tnt  laulrtt'  oroeaduft  for 
conditional  eonfldtnct  Interval*  on  puantUai 
for  a  loeat lon/tcale  family.  Tht  fa*1ly  chotrn 
hart  It  aitrana  value.  Data  sa.y  ba  Type  It  etntorad, 
Nota  that  conditioning  on  tha  a'nclllarlat  glvat  tht 
equivalent  of  an  HPD  region  for  a  nonlnforaat 1  at 
prior. 

Ispllelt  double  ortcltlon  <«-h.  o-i) 
pa ran  alar  <1na«  *  530) 

eharactrr*12  flenitie 
d1»er»1on  i(lnai) 

eoaaen  /dat/  a 

toiaacn  /ea/  enora,*  tuna*  n ,  k 
common  /eb/  c*  oa»<  wo*  t 

common  /ed /  tol 
data  ona  /l.du/ 

data  corn*  fine*  tot  /.t.ri*?*  l,o-7*  l.n-5/ 

Cutout  unit  number  ana  fllenane 
w  r i t  a  (6*  *  >  'Output  unit  nunbtr  T ' 
rtao  <5*  ♦  )  lout 
11  (lout  «na.  6)  than 

write  <4*  *>  'Ml anam*  T ' 
read  <  5*  *  <  a  1?  ) ' )  11  anna 

ot  a  n  <un1t«1out*  11la»llanma,  tt at u i» 'na« ' ) 
and  1 1 

--  T n  1 1  oroora<t,  It  fatten  with  ranon-  data. 

It  car  alto  ba  utad  for  data  fro-  a  Hit. 

Tnr  first  raeord  of  t«*  Inout  flit  hat  in 
temple  tilt  and  t'na  nuaoar  ol  uncentorei  valuft. 
write  U*  *)  'Entar  1  lor  data  Irrr  fit*  ' 
write  (6*  *)  '  0  for  ranaoi  oat*.' 

rtaa  (5.  e )  loot 
11  (laat  .aq.  1)  than 

write  <4*  a)  'Filename  7* 
r  a  a  d  <5*  Mali')')  flmna 

OC  ar  ( ur,  1 1  ■  1  ou  t  ♦  1  *  aile»11enir.e*  itatut>'olr') 
ra ad  <1out*l*  *>  n,  . 

--  hotel  tha  first  flrld  on  ate'  remaining  races  it  a 
batch  Indltator  not  utao  lor  thii  orejra*. 
do  lfi  1«1*  k 

raad  <1out*l*  a)  dummy*  «  (i) 
eont Inut 


call  dtvrjn  (k*  a*  « ) 

ta 

write 

<4. 

a)  ' Sttd  T* 

raad 

<5* 

*  >  Iteed 

ur  1  t  a 

<4* 

a  >  'Ualbull  » h * n a  are  tc.lt 

raad 

15* 

*)  1  bp »  tc l 

ur  1 1  a 

(6* 

* )  '  Semple  1 1 » a  7 ' 

raad 

(5. 

* )  n 

ur  1 1  a 

( 6  / 

a)  'huober  uneaniorro  7' 

raad 

<5* 

a  >  k 

A1 
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— ■  C*t  tha  piaudo«randoa  itapla 
call  rmat  M iaad> 
catt  dr  nu  t  b  <n#  ihp#  *) 
call  divrgn  In,  i#  ■  > 
do  20  4*1#  It 

i  (t )  ■  ad  a*  (1) 
cent Inua 
and  M 

—  Gat  tha  WatUul l  HIE '  t 

call  unrala  (**no#  aid#  n#  k#  n#  apt#  ttar#  100) 

mm  Eitraoa  uatua  location  (mu)  and  tcala  (>19)  attiaatai 
aau  ■  dlog  (aid) 

Rig  ■  ona  /aihp 


--  Wrtta  out 
urtta  (lout# 
urtta  (tout# 
wrlta  Mout# 
It  Moat  ,na. 
urtta  Mout# 
and  1 1 

wrlta  Mout# 
ur  1 1  a  <  lout  # 
It  (idat  »na» 
ur  1 1 a  Mout # 
and  1 1 

ur 1 1  a  ( tout  # 
urlta  Mout# 
urtta  Mout# 
ur 1 1  a  M out « 
ur  1 1  a  Mout# 
urtta  <  tout  # 
lornat  <*’<10. 


uhat  ua  have  id  tar. 

« > 

*>  'Tha  Laulan  conditional 

*) 

1)  than 
*>  'Saad 

•)  'Saaola  ilia 

*)  'Nunrar  uneamoras 
1)  than 

*)  'Walhull  ihana#  icalr 

«)  'Ualbull  tiLEi 
*)  Hitraat  valua  location 
•  )  • L a t raaa  vilua  ittla 
a ) 

a)  ' »atbul l  aata  t* 

1  n u )  (rM)#  1  ■  1  #  k> 

4) 


p  roc  adur  a  ' 


1  '#  tiaao 

t  • »  n 
i  '#  k 

I  •#  ihp#  ic  l 
I  '#  a j  nr  #  aid 

:  '#  Ritw 

1  '#  r 1 3 


--  h  a  ■  #  ealeulata  tha  t  0 l a  r  area  I  i  *  1 1  uitng  br  \ r  *n» 
aaact  atthoa  ana  utCtccic'i  iporoi taut t on . 


urtta  (6#  *) 

urtta  (6#  *)  'touar  eonttoanee  bound  c a  l  Cu  l  *1  l  on  ' 
urtta  (6#  a)  'Guantlla  ?• 
raao  15#  a)  p 

urtta  (6#  a)  'Contloanea  coatflclant  T' 
raao  (5#  a)  qa» 

«-  Lawlati  conditional  rrocrdurr  ( T»c hnoaat r l e  •  #  i#7I) 
call  laulai  (•#  n#  k#  p#  <ja»i#  atol) 

-*  DtCIccto'i  approi  l  aat  t  on  (Tachnorat  rl  cl#  1  s  7 ) 
call  lauaoa  <1#  n#  k#  0#  gar#  atol) 

•*  Wrlta  out  tna  tolaranc*  l  t  •*>  1 1  ratulti 


urtta 

Mout  # 

a) 

ur  i  t  a 

( tout  # 

a) 

•Louar  conMrtanca  tound  on 

a  O  u  1  n  t  1  l  a  1 

urtta 

( tout  < 

a ) 

*  (Tatra a, a  Valua  an*  •  a 

t  0  u  l 

l ) ' 

ur  1 1  a 

(tout# 

a) 

•ProbaM  l  1 1  y 

t  •# 

p 

urtta 

(tout# 

a ) 

' C  onf  1  dan  t a 

l  '  » 

gi" 

urtta 

M out  # 

• ) 

• f  1 1  r ana  valua  quirt l l a 

t  • 

tv 

ur  1 1  a 

Mout  # 

a ) 

'laulaii  tolrranca  limits 

1  •# 

atol#  1 0 ? ( a t  j 

urlta 

<  tout  # 

a) 

'  Approa  t  aat  tom 

«  •# 

itol#  l e  j ( a  * p 

A2 
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ur4ta  (*#*)  2»pdf  ?• 

r  tat)  (*#*)  4typa 
4typ»  ■  1 

uHta  (*#'*)  “tin  and  mat  for  abaefiia  ?' 
road  (*#*)  qm4n#  om 

writ#  (*#*)  *“4n  and  «a>  for  erdlnata  tO#A  for  -itfault]  ?' 
road  (*»*)  oi»4n#  on,*« 
cad  1  nf 1 1  < 960 > 
call  b4nt 1 1 

cad  comm  (4 r ttai  <  13 > #  dm  4  n  3 
cad  conaat  (4baaa»(12>#  o*u) 

If  (octal  »na.  iaro3  than 

cad  coatat  <4baaay<ll>»  or4n> 
cad  eoattt  (4ba»ay(12>#  ona»> 
and  1  f 

4f  <4typ»  .aq.  1)  than 

call  npta  (nolot  ana) 
call  chock  (quant#  cdf) 
call  npt  a  (ng) 
cal t  diolay  (quant #  cdf 3 
do  30  4»1#  nplot-J 

cad  cplot  (quant  H*nqal3#  cdf  <t*nq*13) 
cont  4  nut 

alar  44  dtypa  .to.  23  than 
call  npta  (nnlet  *nq3 

call  chock  (guant#  dani) 

call  npts  (ng) 

call  diolay  (ouant#  oam) 
do  to  4»1»  nplot-J 

call  colot  (guant  (4*ng*13#  oana  (4ifnc*l  >  > 

cont  4  nut 
ano  4  f 

cad  novate  (C#  100 0) 
cad  anaoda 
r  a  a  o  ( * » *  3 
go  to  2 

and  4  4 


urtta  (*#*)  •  <*u4 1  T  • 
road  (*# • (al > ' >  ana 
4f  (ana  .no#  'y' )  go  to  ? 
cad  f  4n4 1 1  (0#  7C  0 > 

atop 

and 


A3 
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program  prtpif 
Hark  Vangal#  July  1985 


c  Program  to  catculata  and  plot  tha  poatarlor  of  a 

e  pareantlla  for  tha  balPull  modal.  Thl»  program  cilli 
c  aubroutlna*  fro*  tha  Taktronfi  Plot  10  library, 
e 

eharaetaraiO  4t'naa 
character*!  ana 

•dlmtntlon  a  <1000)/  eat  (5000#  dam(  5000)r  quant  (5nM)/ 
$  (point  (100) 

data  taro#  ona  /0,d0»  l.dO/ 

c 

(point  O')  ■  0 
nq  ■  100 

nplot  ■  0 

e 

e  Loop  ovar  alt  fttai. 

urlta  <*#*)  '0«er(at  output#  l»tomrlata  outout  ?' 
raao  (*#*)  tbrtaf 

1  continue 

urlta  <*#*>  'Mlanama  T' 
raad  <*#'(a20>»>  flanma 
(f  (flanma  .aq.  '  *>  90  tn  ? 
nplot  ■  nplot  #1 
opan  (unlt«10#  ( 1  la«( l anma  ) 
raad  (10#*)  naamp#  noh* 
do  10  1*1#  nooi 

raad  (10#*)  ldum#a  ((point  (nplot)  at) 

1j  contlnur 

cloat  (10) 

(point  (nptotal)  ■  Ipolnt  (nplot1)  anohl 
e 

e  ••  Calculatr  ina  noatrrlor  (op  a  »p»elfitd  ouantlla 

urlta  (*#•>  'Par cant ( l a  T * 
raad  (*#•»)  n 
It  <0  .qt  •  ona)  e  »  p/l"l 

urlta  (*#*)  'ttanqa  of  valuat  (or  oottarlor  *' 
raad  <*#*)  gain.  qp»« 
do  ■  (qma «-o* ( n>  /  (no»l . > 

urlta  (*.*)  •  Poitartor  o(  •  » 1  #  •  e*  rc  a*t  t  ( •’  ' 

urlta  <*#*)  *  (rot.  '.pain.  •  to  Q»a» 
ur It*  <*#*) 
dc  ?C  (»1#  nj 

(o«  ■  (nplot  “l)*nc  ♦( 

outnt  ltd*)  ■  (l«l>  ado  ♦oitln 

tall  lauap*  <•  ((point  (nrlot )  all#  nu»o/  roft# 

1  quant  (loi)  #  p#  cd(  Md«)> 

(f  ( (or (a  1  . aq.  1 >  t han 

urlta  <*»*)  quant  (ld«)#  ed4  (ln») 
and  ( ( 

i 0  tontinga 

go  to  1 
c 

t  «*  Sou  plot  t  na  raiu 1 1 » 

2  contlnp* 

urlta  <*#*)  'Plot  a  ?* 
raao  <  <■ » '  ( a  1  >  ' )  <n 
11  (ant  .ao.  ' y • )  than 
c 

c  Until  r  ealculatlont  not  yat  ( rp  l  amant  #  i . 
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«  unit*  I*#*)  }*p4f  T* 

c  rod  (•*•>  Ityp* 

Ityp*  ■  1 

writ*  (•*•>  ‘Kin  and  m  for  abaclaia  ?' 
rod  (•*•)  qaln#  qoaa 

wait*  <•**>  »Mn  and  aai  for  ordinal*  [Q*0  far  default]  T' 
rtad  (•*•>  coin*  ooaa 
call  f nit t  (0*0) 
call  blnltt 

call  coaaat  (Ibaiai(ll)*  pain) 
call  coaaat  .(Ibaaaa  <  1 2  >  #  qaai) 

If  foaai  »ne.  taro)  than 

call  coaaat  (Ibaaaytll)*  coin) 
call  coaaat  (1bacay<12>*  tin) 
and  If 
c 

If  (Ityp*  .aq.  1)  than 

call  npta  (nplot  onq) 
call  chacti  (quant*  edf) 
call  npta  (no) 
call  daplay  (quanta  cdf) 
do  SO  1*1/  nplot-1 

call  cplot  (quant  <1*nq*l>*  cdf  (tonq«l)> 

0  continue 

ala*  If  (Ityp*  «*Q.  2)  than 
call  npta  (nplot  *nq) 

call  check  (quanta  dent) 

call  npta  (nq> 

call  daplay  (quant*  darn) 
do  40  1«1*  nplot-1 

call  cplot  (quant  (1*nq*l)«  dam  (lOnqal) > 

40  continue 

and  If 

call  aovaba  (0*  1000) 
call  anood* 
read  (*#a) 
go  to  2 
and  1  f 


urlt*  (•*•)  'Suit  7' 
road  (*» • <al>  • )  ana 
11  (ana  .na.  ' y ' )  go  to  2 
call  finite  40*  700) 
c 

atop 

and 
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progri*  lauiia 

e  Hark  Vangal#  10/U/H 

c  Progra*  to  tost  by  limitation  an  aooao » 1  »at 1  on  ot 

c  DiCieeio  <19#7)  to  tha  Laulatt  conditional  ppee»dure. 

loplieit  double  praciiion  to»h»  o-i> 
paraptttf  Jimai  ■  500) 

Chapattta*12  flanaa 
diotmion  iliam) 

c 

eoanon  /dat/  t 

common  /tt/  cnopr*  tuna,  n»  k 
common  /cb/  p»  gat#  up*  t 

common  /ed/  tol 
c 

data  coin#*  fina  /l.d-2*  1,«»7/ 
c 

e  “•  Output  unit  numbaa  ano  filename 

writ*  <6#  *>  'Output  unit  nunuap  ?• 
raad  <5»  *>  lout 
If  (lout  tne.  6)  than 

unite  (t>  *>  'Filename  " 
faad  <5#'(al?)')  f  lemma 

opan  tunitmiouw  fitemflenmp>  it  at u t ■  '  na » ' > 
and  If 
e 

c  *•  Gat  timulation  paraiatan 


wr  i  ta 

<6/ 

*> 

•Saad  T* 

paad 

(5# 

*  > 

iieeo 

upi  ta 

(6# 

*> 

'baitull  inapa  an*  tcila  ’ 

raad 

<  5  » 

6) 

ihpp  id 

upi  ta 

u> 

*) 

,r>imple  life  T* 

paid 

a, 

•  > 

n 

up  1 1  a 

<  Op 

•  > 

"unban  unctniorf'1  ’ 1 

Paao 

(l> 

*) 

k 

uplta 

(it 

*  ) 

'bum tar  of  paolicatai  ?• 

paao 

(it 

») 

nil  m 

up  i  ta 

16 1 

*> 

'  u  a  n.  t  i  l  a  t  • 

paao 

( 5  t 

*) 

0 

upita 

<  6  t 

*) 

'Confidence  coe'iicipnt  ?' 

paad 

(it 

•  > 

din 

"•  b  p  i  t  a 

out 

uhat  up  have  in  tar . 

upita 

< i out  t 

*) 

urita 

<  iout  t 

•  > 

•Tha  Liutan  eonditinnat 

c  p  ec 

a  i  u  a  a  ' 

unite 

( i out  t 

*> 

ualta 

( lout  t 

•  > 

'Saad 

l  ' . 

l  kpa  ! 

upita 

(lout* 

*> 

' hunt ar  ot  r»ulicatat. 

i  ' , 

n  «  y  » 

upita 

( 1  out  t 

*  > 

'Samp  la  t i pp 

t  ' » 

n 

upita 

< i out  t 

•  > 

•hupppr  uneemontd 

!  '  * 

k 

upita 

( i out  t 

»> 

' buant i t  a 

1  ', 

[ 

upita 

( i out  t 

*> 

'Contidanta  coefficient 

1  '  t 

V  *  " 

upita 

( i out  t 

*> 

•uaihutl  i  n  a  o  e «  ical» 

t  '  t 

t  ►  ?  #  *  c 

upita 

( iout  t 

» ) 

»•  Loop  over 

tna  pi o'  rirtieatai 

call  r n»a t  (utto) 

I  ■  C  .  d  0 

II  »  O.dO 

do  95  l ■! »  min 
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»  c»at  tdt  6»#udo**r»n1or  »a*>t>l* 
tall  drnwko  tn#  »hw#  »> 
tali  dtvton  <n»  «*  «> 
ao  2C  k'i*  *  ... 

,  M)  •  atl  *»  <k> 
tent iriut 


-  U-l»»»  tonaUUn.l  oroc-flur*  l  Tac  *  nor  at  r  <<  »  * 
call  laalat  < »'  n»  »>  «**'  f  1  1  ’ 


(KUdo't  aocroititat  ton  (Tachnonat f  *  e 1#  1C| 
call  la»ae»  <»*  ft  n'  ?*w*  atoll 


.  datura  .eero.U.ttof  <n  tjrM  «<  carc.nt  "tor 
dcet  ■  (atel  -atol)  /atol  *10f. 


..  irH«  out  tha  tolarinea 
artta  (tout*  *>  'liulati 
a  t  o l *  at o  l  >  ape t 
tat  ♦occt 
it  ■  ii  ♦at'tt  oiset 
cont  t  nua 


Mutt  ratutt* 

,  «tf teetoa  nalt  a 


mm  Kf«n  ana  itan 

l  a  i  /min 

ii  ■  »ort  <<tt  - 

«rtt»  (lout#  *»> 
aftta  k  tout#  l!,i  * 
a  r  1 1  a  (tout#  *•  >  * 


•  orv.  at  t«  aft  ft 
ni  t »  *i«i>  /  <nt  k* 

♦  v a r a t »  after 

Stops'*’4  lavtatktf 


v  (  *  i  k  ••  »  *.  • )  )  k 

I  1  *  t 

*  t  *  *  i  ’  1  '  •  *  '• 


It  oc 
anfl 
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ubrout In*  lawlti  (a*  n a#  ki>  pi,  pama*  ttol) 

Mark  Vangrl*  May  1998 

lubrouttn*  •taul**'  caiculatta  on*  »ld»d  low*r  toltrane*  limit* 
for  th*  Mtibull  *od*i  uilng  t h»  La«l**i  eandltlonal  proearlur*. 

T h 4 »  reutin*  o*rform»  '*aaet'  calculation*  by  numerical 
lnt*gratlon.  for  *w*n  oodorattly  4are*  attaoltt*  th*  hlClcclc 
•oeroalpat ion  uttd  In  tubnoutlnt  'lauapa-'  It  vary  aecunt* 
and  coaputat  lonal  ly  l»»»  troublatoat  than  th*  »*thed  u**ct  nrr*. 


* 

n 

k 

P 

9 

•  tot 


Oat  a  ( Input ) 
Total  *a*pl*  lit*  (Input) 
Mu»b»r  of  (unc*n*or*d)  ob»*rvation*  (tnru*) 
Probability  atioclatto  with  auantll*  to  < !  n  r  u  * ) 
Conflo»nc»  l*v*l  for  Ice  on  an  (Innut) 
'Eaact'  iou*r  toloranc*  limit  (Cutout) 


htf  i  L  a  a  1 1 » i »  T*ehno**t rl ct  1978 

lopHclt  doubl*  or*cl»lon  (a«h#  o*i> 
dim*n«lon  a  (1) 

common  /par/  tiho*  ttel*  amu#  aie 
common  /(*/  cno r »,#  tum*#  n ,  k 
common  /eb/  p>  ;*n#  wp<  t 

common  / c d /  tol 

*at*rnal  antpnd*  aconf 

data  coarctr  #o*»  fin*  /l. •■)-?*  l.d-5*  l.d-7/ 
data  i«ro*  on**  c.lr  ch  /f,<ig#  l.dbr  .tdD*  1  •  ?  c  C  / 

Put  atuff  In  common 

n  »  na 
k  •  ki 

p  ag* 

pan  ■  pam* 

"•  <i»t  t  h*  Wtloull  *4Lf,'l 

call  unrml*  <*»ho>  «icl>  n ,  l>  i>  *p*»  it*r»  in;) 

«“  Tol*r«ne*  limit  factnr  rano*.  T n 1  *  r*"o*  i  «  n'rao 

tnouor.  for  virtually  any  practical  confidant*  eo«iM[u»*  , 
•Itonf*  provlami  a  first  any r o» l * « t i pn  to  th*  t«l*rane* 
limit,  t.otm  that  Mconf*  rmturri  lourr  toltrmcr  limit*. 
9 am  »  on*  ••()**■ 

call  Iconf  ( * 4  n ,  k,  p,  oamr  tollmt) 

aam  ■  on*  •  q  a  r 

tollmt  ■  log  <(*»cl  /tollmt)  ***ihp> 

I  l  ■  Cl  *tol Imt 

*h  ■  th  *tol Imt 

writ*  ( * / * )  *  L *u I m  i  First  gut**  ■  ' ,  loo  (tollmt) 

«•  how  go  to  to;  teal*, 
do  IP  1*1*  k 

>  t  U  ■  log  (  I  (i)) 
cent Inu* 

“•  E  a  l  r  »  a  *  walum  location  (itul  any  teal*  (am)  *  *  1 1  r  a  t  <  * 
i«u  ■  log  (tit  t ) 
i*o  ■  on*  /**ho 
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e 

«  »«  W»  naad  th a  anelllarlat  and  th*1r  iu* 

au»a  ■  itro 
do  20  1«1 ,  k 

ium  ■  tuna  ♦<•  ( \ )  «i*u)  /*ig 
20  eontlnua 
c 

C  ■"  Kta«t  aa  obtain  tha  eonttant  o<  Integration, 
c  Uta  an  adaptive  quadrature  routine  to  Integrate 

e  XRT6NL'  on  <0,  Infinity), 

enorm  ■  ona 
aoi aff  ■  taro 
ralarr  ■  apt 

cell  dqdagl  («ntpnd,  lere#  la  aUarr<  ralarr#  a  r  r  > 
c 

e  --  Tha  neritallalng  eonttant 

cnorr  •  ona  I  » k 
art  ta  <6#  *  > 

writ*  <0#  'Tna  norpallilno  eonttant  It  '#  eror* 
e 

e  *•  pth  ouantlle  tor  ttenoard  entrem*  value  dlttrlnutlon 

w  P  ■  1 0  9  t-loj  <  on*  -pi) 

e 

e  fitlaatad  ot h  quantlla  lor  data 

ip  a  afty  mg  *. n 

c 

e  «■  frant't  algorlthit  to  tine  tolerance  H*1t  lactrr  tue” 

e  that  tna  Integral  ot  T'JTPiO  Iron  II#  IntiMtM  aiualt  'tf'. 

e  Tha  tint  natt  hat  a  lama  arrer  tolararea  to  lave  1 1  m . 

c  Tha  taeony  patt  ui»t  tna  tlnal  tolaranca. 

c 

a  a  a  1 1  ■  hi 

abtar  r  a  j»rc 
ralarr  a  eoarta 
1 ol  ■  eoarta 

call  Oibran  <«eonf,  actarr,  ralarr,  r I ,  rr,  taiit) 
e 

e  "*  hove  In  tor  tna  Mil, 

an  It  a  ICC 
I l  ■  ,950v  a** 

It'  ■  1  i  0 5 q *  *lh 

ralarr  a  t  tna 
tol  »  Una 

call  oibran  (icon!,  altar  r ,  ralarr,  *1,  r  >  *  a  M  M 
t  o  1 1 1 1  ■  i  h 

c  "■*  Calculata  th»  tolaranca  11*11 

•  tol  ■  f«c  ( a  hi  u  *  i  h  a  i  »p  ) 

c  --  Rettore  the  data 
do  50  1  •  1 ,  k 

«  (1)  ■  tip  (i  <1>  > 

Tu  contlnua 

raturn 

♦  nd 
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e 

c 

e 

e 

e 

c 

e 

e 

e 

e 

e 

e 

c 

e 

e 

e 


e 

c 


c 

e 

e 

c 

e 

e 

e 


c 

c 

c 

e 

e 

c 


e 

c 


aubroutlna  lawapx  <  x.  neampat  nobaat  pat  pama.  atol) 
Mara  Van f*l<  July  1999 

An  excellent  approximation  to  the  ona-eided  Melbull 
conditional  tolaranea  llmlta  of  Lawleea  (1979) t 


x  —  Data  (Input) 

naanp  —  Total  aample  alia  (Input) 

noba  —  Number  of  (uncenaored)  obaer vat  lone  (Input) 

P  —  Probability  aaaoclatad  with  quantile  xp  (Input) 

«  ~  Confidence  laval  far  led  on  xp  (Input) 

atol  —  approximate  lewar  tolaranea  Holt  (Output) 


Ref  I  Lawleea.  Technometr lea  1979 
Dleleclot  Tachnoaotrics  1997 

implicit  doubia  praelalon  (e-ht  o-») 
dlmanalon  x  (1) 

common  /par/  uahpt  uaclt  xmut  xa« 
common  /eonf/  pt  famt  naampt  noba 
axtarnal  aeenfl 
data  maxlt  /100/t  ape  /l.d-B/ 
data  coaraot  fine  /l.d-3.  l.d-7/ 
data  aarot  ona  /O.dOt  i.dO/ 

data  elt  eh«  dl.  dh  /.OdOt  1.2dO.  .99d0t  l<09d0/ 

—  Rut  atuff  In  eommon 

f  ■  M 

•am  ■  ona  -tama 

noba  ■  nobaa 
neamp  ■  neampa 

—  Hat  tha  MLK*  a  of  tha  Walbull  paramatara 

eall  wnrmla  (uahpt  uaclt  naampt  nobtt  xt  apa>  Itart  maxlt) 

—  Tolaranea  Halt  factor  ran#e.  Thla  ranpa  la  broad 
anoufh  for  virtually  any  practical  eonfldonee  coefficient. 
* leonf*  providaa  a  firat  approximation  to  the  tolaranea 

Halt. 

eall  leonf  (x.  naampt  noba.  pt  pam.  tollmt) 

*1  ■  el  atollmt 

•h  ■  eh  atollmt 

write  <•«•)  *  Lawapx  I  Firat  iuaaa  •  *.  tollmt 

~  brent’a  alporlthm  to  find  tolaranea  limit  factor  at 
tha  daalrad  confidence  level.  Tha  firat  paae  haa  a 
lar pa  error  tolaranea  to  aava  time.  Tha  eecond  pace 
uaaa  tha  final  tolaranea. 

maxlt  ■  100 
abeerr  ■  aero 
relerr  ■  coaraa 
tol  •  coaraa 

call  dibren  (aeonfl.  abaarr.  raiarr.  alt  ah.  maxlt) 

—  Hove  in  for  tha  Hill, 
maxlt  •  100 

al  ■  dl  «ah 
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c 


■  dh  «ah 
Imrr  ■  fin* 

1  •  ♦in# 

li  dibran  (acon#l<  abmarr.  ralarr.  *1.  «h,  Mxit) 

•t«I  ■  «h 

rat urn 
and 
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subroutine  Iconf  Is,  nsamp,  nobs,  r ,  g,  toilet) 

Hark  Vangelr  Oetobar  1VF6 

A  non-iterat 1 ve  first  too ro* 1»et Ion  to  the  Lawless  ennditirral 
procedure  lor#  el t arnat 1 v a l y •  to  tha  posterior  of  a  quantll*  under 
a  Mat  oHor).  Tna  routine  Is  written  for  two  paraater  W  a  1  d  >j  1 1 
analysis#  but  attention  to  t  hr  aenerallttd  too  ganirs  family  It 
at raight forward.  This  routine  returns  the  estlaated  eonlldencr 
llolt  for  a  provided  orootblllty  level  and  eonfldenea  (hence 
tbe  '1*  "  for  Inverse  “  In  tha  routine  naira).  Tnls  routine  it 
aperns iaata ty  Inverse  to  'aconf.  It  provides  the  'care  result 
as  ‘lawaps'  out  with  a  lets  accurate  aperoslratlon, 

Hal  \  Dlelcclor  T.J,#  Teennowet rl cs  19*7  p.33 

s  --  Data 

ntanp  •-  Total  sample  tlie 

noCt  ••  Number  of  (unc »nsor»d )  observations 

g  •  »  Conflaance  level  for  tcb  on  «p 

p  •  •  Probability  associated  with  auantlle  so 

toilet  —  Upper  tolerance  Unit 

Implicit  double  oreelilon  (e-h#  o-e) 
di pension  «  ll)r  s  ( A  > 
data  ecii  resit  /l.o*5#  100/ 
data  aero#  oner  halfr  thhait,  fvhall 
1  /O.ddr  l.qOe  O.J-nr  1  .  *>  d  f  r  J.JdO  / 

-•  Get  the  welsutt  “lr,s 

call  wnrnle  lusho#  usclr  nsa»i'/  no'sr  t,  er  t>  iter,  *.  t  *  > 

--  T ranilorp  to  e»tre*e  value  di strltut Ion, 
uloc  •  log  <usct> 
uscl  ■  one  /usno 

--  Calculate  the  derive* ives  ot  t  *■  e  log  1 1  k  »  1 1  n  o  o  » t 
the  “IF. 

wo  ■  log  < -l og  < one  -p )  > 
do  SC  j  *  1  r  4 
s  <  j )  ■  ae  r o 
do  4C  1*1#  nobs 

i  ■  ( l og< ■ ( 1 > )  -  u  I  o  c )  /uscl 

t  ■  <  (a  »wp)  ae  j  >  (j ) 

s l j )  ■  s(j)  *t 
cont i nve 

Slj)  ■  s  <  j )  elnsamp  -robs)  *? 
conti nue 

diO  »  -nobs 
dll  «  -til) 
dO?  ■  -loots  m  l  ’  >  > 
c  3  C  ■  no  t,  s 
d 2 1  »  ?*nohs  Mil) 

012  ■  3*S  <  1  )  Ml?) 
dOS  •  nobs  *3e»l?>  *i(S) 
d4C  ■  -nobs 
d3 1  ■  -l Jenot  s  ♦ s  ( 1 )  ) 
d22  »  -uanobs  Mesll)  ♦  *  C  ?  )  ) 
dl t  •  - <  7*»  1 1 3  ♦ 0*  s  l  ?  I  M<»)  > 

804  ■  -nobs  -(Test?)  *b*s(t)  *sl4>) 


l ! nnut  ) 
l Inout  ) 

( I nrut  ) 

<  I  n  o  u  t  > 

1 1 nout ) 
lrutput ) 
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--  Th*  appro»1aata  atari  a no  varlan ca  el  r#  tha  *1onto 

•quart  root  o  1  -2  tloa*  llkallhooo  ratio#  It  calculate  1r. 
taro*  ot  tha  dlj. 
all  ■  -d02  / < d ? 0  *d02  -dll  *dll> 

a  •  (vll  **ha l 1  *<-dl?/dO?  *dll*oC3  /d02**2>>  /? 
b  ■  (vll  0(-d22/dQ2  ♦  <  d21 *d03  ♦2*dl3*dll  adl?**2)  rd P2»*> 

t  -<4*dl2*dll*dOJ  *dO(*dl  1**2)  /d02»«! 

t  ♦2*(dll*d03  /d02**2 )  «»2>>  /* 

c  ■  (vll  oothhaH  *<d3G  -Jad21*dll  /d 0 2  *3adl?  o(dll  /d0?>  **D 
%  -d03  *(dll/d02)  **J>>  n 

d  •  (vll  402  *<040  -<4*dSl*dll  *3*d?l**?>  ✓  d C* ? 
i  ♦(6*d22*dll**2  ♦l?*d21*dl?*dll>  /d02**2 

*  -<6*d21*d03*dll**?  a4*dl S*dl  1**3  ♦l?*<dir*dll>**2>  /a *2r<.» 

i  ♦<12*dl2*d03*dll**3  *d04*dll**4>  /dfl»**i 

S  -3*dl 1**4  *d03**2  /d0?**5>  >  /24 

rou  ■  a  «?*c 

ng  ■  tart  (onr  ♦2*<b  *3*d  ♦a**)  ♦ll*e**2) 

-•  OlCIttlo#  aquation  b. 
ig  ■  dnorln  <g> 
rg  •  iou  ♦  *•#  *ia 

ye  •  uloc  ♦  u  t  e  l  *  *  n 

toll  at  ■  ye  ♦u»et  *<tqrt  ( v 1 3  >  *Crg  *c*ro  **?  * 

1  (Ivhatl  *e  **it  ad)  *r#  **3>> 

--  Go  to  Wtl Dull  tea l a. 
to  l  lot  ■  a«p  (tot tmt  > 

rrturn 

and 
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doubt*  prtefsfon  function  antgnd  ( r ) 

Proportional  to  th*  pdf  of  •  e*rtain  pivotal  Quantity 
Th*  nermaliting  eonatant#  'cnor*'/  d*p*noa  on  th* 
dot*  and  muat  b*  obtain*o  by  a  or*liminary  numerical 
int*gration.  One#  ‘enorm*  fa  know n*  b*eo**a  a 

pdf  and  la  gatd  ey  YhT&ND  ourino  th»  primary  humMmI 
int*gration  to  o»t  the  toltranc*  limit. 

Not*  :  •cherm*  mutt  b*  initialitad  (to  1)  b*for*  th* 

preliminary  integration.  Following  that*  'enor*' 
tan  b*  aaaigneo  th*  valu*  wHeh  makaa  XMith*  a  ra< 

implicit  doubt*  or*cfa(on  (a-h*  o*f  > 
dim*neion  a  ( 1 ) 

•»  *al*  la  ua*d  by  VNT&Nfi*  h*n<*  th*  common  I  lock, 
common  /dat/  a 
common  /ec/  al 
common  /  e*/  enor**  a  urn  a*  n*  i> 
common  /par/  *ann*  * s c  l  *  a*u*  aae 
data  i*ro*  on*  /A.10*  l.oO/ 

pur  ■  on*  /float  < ► - 1 > 

al  ■  tare 

t  ■  on* 

do  10  i"l*  » 

ft  <i  • *o ■  k )  t  ■  dhl*  (n  -*  *1) 
a*  <*  <  f  >  •**u>  /  a  a  ,j 
ll  ■  al  *t  mono  (a  m»  *uew> 
eont 1 nu* 

antgna  ■  lal  /dul*  t  k  >  >  **t-k>  *rwr 
antgnd  ■  antond  m*»w  <iu"*  » ( *  *>t»f  -on*)) 
antyra  ■  antona  *cnor» 
r*tyrn 
•  nd 


douolf  oracfaion  function  »conf  (to) 

e  Subroutine  to  datarmfne  t  h#  orobacflfty  Of 

c  t  h»  pin  quantile  bainq  ton  thin  T(t)/  where  T  ft  t  k ► 

c  tolaranc*  li*it  fictor, 

fioUcft  doublt  oraeftion  <a>h»  o"if) 

COiMiOn  /  e  t  /  p»  gar.,  yp.  t 
cornnon  /ed/  tol 
aatarnal  yntflnd 
data  tarot  ont  /0«du*  l.do/ 
c 

e  ••  *t*  ia  to  b»  founo  an  that  the  integral  of  V  *  T  t  > "  tr»« 

c  <0,  infinity)  equal  a  '  nait. •, 

t  ■  ta 

abaarr  ■  tol 
relerr  ■  taro 

call  oqaagf  (yntgnd,  ter o,  1/  aottrr,  ralarr,  >1,  arr) 
aconf  ■  if  >31111 
c 

c  --  Hia  fa  j u a t  tone  terminal  outrut  to  amute  tha  uttr  mil' 

c  ha'a  waitfn*  for  raaulta. 

wrfta  <*»*>  '  Tolaranea  factor!  confluence  «  t,  if 

rtturn 

ana 
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c 

t 

c 

e 


c 

e 

e 

e 

t 

c 

e 


doubt#  praelaton  tunet4on  yntgnd  t*) 

Junction  to  caleulat#  th#  Integral  lor 
dataralnlng  tha  tonlldenee. 

4»pl4ett  doubt#  preclaton  (»•►>'  ®**> 
coaaon  /eo/  p •  g»*r  uo#  4 
coaaon  /<■•/  cnor##  au*a»  **  * 
co.aaon  /ee/  »  ,  ..  . 

data  *ero»  on#  /0»d0»  l«dOr 


par 


■  on#  /Moat 


—  Th*  f  4  rat  factor  4i  to#  (unction  uMth  «»»  tnt#orat#a 
to  gat  'enorn'. 
yntgno  »  »ntgnd  <i> 


--  The  iteoro  4  actor  la  th»  evaluates  at  « ' 

uhlch  depend#  on  a#  a  auantlty  caleulat# 
upHc  ■  a  *de«P  (up  #t  **  **pur> 
a  ■  on*  -dgamdl  tlloet  U><  uolliO 

untsmrt  ■  ant  And  *C 


r#t  um 
and 
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double  precition  function  econfl  Ct > 

ioollclt  double  oreeHion  <e-n#  o-*> 
e 

dioenelon  * C 1 > 
cooeon  /det/  * 

coneon  /eonf/  p *  ge"1*  n»»«P*  nob* 

cell  eeonf  <«»  n*»np#  nob*#  t#  p#  rrob) 
•eonfl  ■  prob  -gee 
c 

return 

end 
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tubroutlne  aecnf  <a,  ntifp,  nohi,  q.  c»  tonU 
Hark  Vangel ,  July  l’b* 

An  appro » 1 me t 1 on  to  the  Lawlett  conditional  procedure  (or. 
alternatively,  to  the  poiterlor  of  a  quantile  under  a  flat  prlnr). 
The  routine  la  written  for  two  paramter  Velbull  analytic,  but 
attention  to  the  generallteo  log  gamma  family  la  etralqht f orwar n. 

Be f  i  hlcleclo.  T.J.,  Tee hnomet r 1 ea  19*7  p.?? 


t  Data  (Input' 

niarp  «  Total  tamole  tire  (Input) 

notot  -»  Munoer  of  (untentcred)  ootervatlona  ( I « r.  j  t  > 

q  —  Value  tor  which  P  (*p  ■<  o)  It  detlren  (Input) 

p  --  Probability  ataoelater  with  ouantllr  >o  (Inrut) 

eonf  »-  P  (xp  ■<  a)  (Output) 


Implicit  double  orecltlon  <a-h.  o-» ) 
dimention  «  (1).  t  (4) 

common  /par/  uthowr  utelv,  <«uey.  atoev 
data  apt.  matlt  /l.d-5,  100/ 
data  tero,  one*  half,  tnhalf 
i  /O.oO.  1.80,  ,5u0.  1.581/ 

--  fret  the  conctralned  MLff,t,  where  o  It  eonttrilne"  ic 
be  the  pth  quantile. 

call  ewnr*,l  (p,  a,  etho,  ctcl,  ntamo,  nor*,  a,  art.  1t»r.  - m »  It) 

-*  Trantfor*  to  eatreme  value  dl tt r 1  cut  1  on . 
uloc  ■  log  (utelw) 
e l oe  ■  loo  (ctcl) 
utcl  •  one  /ut ht>» 
ctcl  ■  one  /emr 

»•  Calculate  th*  » 1  a n •  .1  tqu»r»  root  of  -?  tinea  the  tv. 

of  the  llaellhooi  ratio 
wL  e  log  <  -lOJ  (on»  -C ) ) 
el  ■  rero 
u l  »  rero 
oo  20  1«1»  not » 


t  ■ 

log  (a 

( 1 )  > 

c  l  ■ 

cl  ♦  (  t 

»c  l oe ) /c i c 1 

•e  ap 

((t  -c l oe )  /c  i c  1  ) 

ul  ■ 

u  l  *  (  t 

•uloel/uiel 

•ear 

< (t  *ul oc )  /uic  l  ) 

cont Inue 

C  l 

■  Cl 

»(ntamp 

-noon) 

*r  a  n 

( (t 

-c loc ) /c » c  l  >  *nst  a 

* l on  i  c  a  c  l  > 

ul 

■  ul 

-  ( n  a  a  m  o 

•noui) 

me  a  n 

( (t 

-U  t  OC  )  /UIC  l  )  *"0H 

* l o  *  (utcl) 

air  •  -*  etc l  —  u l  > 

agn  ■  log(a)  -uloc  «wp  mutcl 

tgn  ■  abi  lagn)  /tnn 

r  ■  ign  eaort  (abt  (air)) 

••  to.  ctlculete  tne  derivative*  cl  the  (o;.  llkellhror  at 
ine  *11 . 
do  30  j«l,  m 
t( J )  ■  terp 
00  40  1*1,  not  I 

i  ■  (lontaHJ)  -uloc)  /utcl 
t  m  ((I  «wp>  **))  mean  (») 
tlj)  »  t(j)  *1 


40 

continue 

»<j>  •  »<j) 

-non*)  *t 

50 

continue 

«* 

d20 

■ 

■note 

dll 

■ 

db2 

m 

-<nob*  ♦*<?>> 

* 

d30 

m 

nobi 

d21 

• 

2* not*  *»(1) 

d  1 2 

■ 

3*1(1)  ♦*(?> 

dbS 

■ 

noo*  *J**(2)  **(3> 

d4  0 

■ 

-not  i 

031 

■ 

-(  jMflOftl  *  *  <  1  >  ) 

02? 

• 

-<  4*"Ol  v  ♦ 3 v* ( !  ) 

*t(») > 

dl! 

■ 

-<?*i(l)  ♦  *(?) 

♦  *(’.)> 

dU 

■ 

-nob*  •(?**(?)  *M*(1)  *»(i)> 

c 

e  -•  Tn*  » pr  ro » t  it  1 1  ♦  »«*n  »no  v*M*nc»  of  r  H  c*l! 

e  tor**  of  t  ft#  of  J . 

vll  ■  -dC2  /(o?0  Mu?  -all  villi) 
e 

•  •  (via  *»hiH  *(-dl?/rtO?  MUM'S  /r*o?«w2)>  n 

t  ■  ( v i j  *(M2?/u*  T  ♦»'i,idi  ?«a:  i  m; 

l  Mu4M  1 1  *1*2  >  /  r;  ^  2  >:■  tn  ? 

1  ♦  if* < oll*d‘. J  /dj?»M)  **?>>  /<. 

e  •  (vll  **th*'#H  *(«*•:  -3M21M1J  /Cftj  ♦»*-:;  *i 
4  -dul  *(dll/il’>  *M>>  /* 

1  ■  <  vll  **?  *(d«')  ft 

1  ♦(#. •Ko?2*all^».>  ♦l.,i!i-;’U'ol?*dl.1  >  /"VipJ 

1  -  <  ft  *  U  r  1  Pi  i  0  1  '!■  0  1  1  *  v  5  *  1  ?  P  t  o  1 1  :  1 1 

*  ♦  (1  ?m1?*103M1  tw-ji  *-v  4*-l  1  «•*« )  mu  /, 

1  -’Ml  \H'4  «l v t  /••  V  ?«  l*  •> )  )  /Ti 

c 

«#u  ■  t  ♦»■<{ 

«  »  d  »  Id  ft  (on*  ♦  i*  '**  ( !  •  i  >*  j  <*  a  m  c  >  +  J  1 1' c ) 

c 

c  -  »  (r  •  ■  i>  u >  /  »  i  j  1  *  •.  p  r  c  >  t  in  t  f  l  >  d  i  *  t  f  f  p  w  t  •  •  '('* 

r  ■  (r  •»*  u  )  /  ■  t  ■» 
t  on  t  ■  dnordf  (f> 
e 

return 

fn: 
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:i  l  ft  t •  :  i  - 

’l‘t'  f  )  .ft  ’ 

-  1  1  /  i  *  ?  )  1‘  I  *> 

If,-*! 
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SUBROUTINE  UNRHLE  (A*  b#  *'SAMR#  NOBS*  <»  AEP5*  JTFR*  *M|T) 
MARF  VANfifL*  NAY  1 9 fU* 

ESTIMATE  ME  I  BULL  PARAMFTfNS  BY  MX1HUN  LIBEL iHOOr 
I  ..  wrjAULL  $NA*’t  PAHAMFTER  (RETPUNFC) 

P  —  NCI  BULL  SCALE  PARAMETER  (*ETU*N!D> 

NOfSP  --  NUMBER  OFZ0»SrRVATl0NS  (UNCFNSCREP) 

(NSAMR-NOAS)  VALUFS  ARE  TYPF  II  CFnSU,l'' 

X  —  DATA  valufs 

AEP5  —  ABSOLUTE  CONVERGENCE  TOLERANCE  (INPUT) 

ITER  --  NUMBER  OF  ITERATION*  RfOUIRFr  <MFTUB’.Er  > 

MA N I T  -*  MAXIMUM  NOKPE0  OF  ITERATION*  (JNR'lT) 

IMRLICIT  DOUBLE  PRECISION  (A»H#  0-Z> 

DIMENSION  X  (1>#  CVTBL  (?S>#  SMPTfel  C  ?  5  > 

--  CCRFFIC1INTS  OF  VAR1ATIO*1  FOB  “O^INT  E$T1mATp  OF 
SHARE  PARAMETER 

DATA  NT bL  /25/  . .  ..... 

DATA  CVTBL  /l.  *  .JOJAS*  '\\\\'' 

I  .19J77*  ,16A02*  .1AB37#  .)3?6A#  .17CJI* 

I  . 1 09A A  *  ,1012?*  .0977**  .nj737*  .  0  fl 1 7 • * 

>  .07067#  .A7?S1*  .06F62#  .06X13#  .061<>'# 

j  ,  OA 1 77#  .031S0*  .02X?«*  .01694#  .D177V 

DATA  SHPTbL  /l. #?. *3. *A.*S.*<. #7. #P.*V.#1G. #11. *)?.#)-.' 

}  14, *1  * • #16. *17, *  1 F. #19. *  2 * • *  JO • # AO . # 1 0  •  i 7  ■>!  v./ 

PATX  2 F K 0 *  ONI#  TCLMlN  /m.OO.  l.r0*  l.r,-5/ 

—  SCALE  THE  C*T» 

SCALE  ■  y  (1) 

rc  io  i«i#  s'to  .  . 

JF  (X  (I)  .ST.  i u  *l  r  >  scale  ■  >  n > 

CUNTIME 

00  2 C  !• X  *  NOtS 

X  (I)  ■  X  (I>  /  SC*Lr 

CunT 1 nuE 

XSAMR  •  MLF  ( NCf X) 

PXSA“R  ■  Of: t  /  a S * “ R 
FPS  ■  AEOS 

IF  UP*  •  E  0,  •  2  r  -a  C »  £®S  “  VH*!1 

--  CALCULATE  S<IM,  SU“  nf  LM*#  AM  S"*  Cr  *  0 1 1  *  '  E  f  *(  C  * T  * 

SI  *  2 C H C 
l  ■  2  CKO 
S b  •  EERO 
PO  SC  1*1#  Nil.  s 

SI  ■  SI  «  LOS  (»  u>) 

S  •  S  ♦  X  (1) 

SB  ■  SS  •  X  <  1 )  *>  ( I ) 

CONTINUE 

*1  •  SI  t  XSAMR 

—  USF  THE  CCEFFlCirsT  nf  v A >  1  A’ |  M  T  a  *cT  **[  T  -  ■  :r 
‘•UMENI*  ESTIMATE  U*  Tbp  r -1 A  M'  PApAMtUr’ 

CV  ■  SURT  (NltS  Rbl  -S  »*>  /* 

IF  (CV  •  SE  ■  CVT3L  (1)  )  lut ' 

A  •  SHPTBL  (1) 

ELSE  IF  (CV  ■ LE •  CVTML  <NT»L)>  THE* 

A  ■  SHPTBL  CULL) 
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ELSE 

DC  40  I>2,  mi 

if  (com  u>  ,ir.  cv>  t“e* 

A  ■  SH*TOL  <  I  - 1  >  ♦  {SW*>TH  (!)  -$H*TH  <I-1))i? 
S  (CV  -CVTH  tin  / 1 C  V  T  i.  L  C 1  - 1 )  -CVTH  <  I )  ) 

sc  to  so 
END  If 

AO  CCINT|«U! 

SO  CONTinr 

E.»c  IF 
C 

C  —  LOOP  III  TILL  CPNV£BSf.Ci  C"  lTrw»TIC"  lpp  rxcEr.m 
1 1 EN  »  Cl 
1  CCMTIK'JL 

c 

C  —  CALCULATE  SU*S  F 1 PST 

S?  ■  2EF0 
S3  ■  2£<  : 

SA  ■  2l»fO 
DP  40  1«1#  NIAS 
T 1  ■  A  CD  Am  t 
T2  «  T2  *  Ur.  (x  tin 
TS  •  T?  *  LuC  Cx  tin 
SI  ■  $?  ♦  u 
L  3  ■  S  ?  ♦  T  i 
*4  ■  SA  ♦  T» 

40  CONTINUE 

52  •  S2  +  T;  •ti.jA-c 

53  «  S3  *  T i  •Cf.SA'-U  • 

54  •  SA  4  T 3  KI.SA'  I  -".*1' 
r 

C  --  CAlCUL»Tt  •  P,  S'JA?>t  I'll'ir 

Tl  ■  l‘E  ✓  A 

t.  •  s’  / 

TJ  •  $4  /  %2 

<1  •  A  ♦  ( T 1  ♦  0)  -  Ti  )  /  I T 1  m T I  ♦  T»  -  Tp.nP) 

C 

C  —  HIT  WmPs  C" 

If  <*-'  <  A  ]  -  »>  ,L  1 .  *t>*  .1*.  PE'  .«T,  M]T)  T  • 
PE"  ■  1T'\  ♦  1 

A  ■  41 

fiL  T(  3 
C 

C  —  CALCULATE  SCALE  F*i»1,rTr-  F  r  l*’  '.iC'  n  * "  *  “  T  ’ 

’v  CCVTJf.  .i. 

"SUP-  »  2F-  P 
M  rc  1  »  in  I.'tec 

051“  •  ♦  *  U)  »"»  A 

7J  CONTI  ML 

PSU-  »  t  M  •*  *  f  L^AT  (NS*  *P-  *>■  (  .*  P)  it "  A 

e  ■  (h*SA‘Pi  ssj“>  *>*  t  /n 
c 

C  —  UN  SC  AL l  OAT* 

f  •  f-  *  SC  ALT 

ro  ec  i * i > 

A  1 1  >  |«  t  t  I  >  *  j'n. 

A  0  C  O  N  T  1  ti  I 

i  i.TU‘  I. 
t  ;.D 
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c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 

c 


c 

c 

c 


c 

f 


10 

?0 

c 


c 

c 


3a 
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SUBROUT  I  Nt  CWN»ML  <P#AXP#  A#d#  F.SANP#  HOPS#  X  #  Arp-,, 


HARF  VHNGEL#  JULY  1981 

ESTIMATE  ME  IEULL  PARAMETERS  BY  MAXIMUM  LIF5LIKCI.  WITH  Hi 
CONSTRAINT  THAT  ThE  ESTIMATED  PTH  QUANTILE  AT  T n T  k'Lr 
IS  EQUAL  TO  XP. 


p 

m  m 

PRuSABIL  1TY  LEVEL  OF  QUANTILE 

XP 

mm 

pth  muantilf 

A 

m  m 

WEI  BULL  SHAPE  PARAMPT**< 

c 

m  m 

wEIBULL  SCALE  PARAHFTFH 

NSAMP 

m  m 

SAf.PLF  SI  2  L 

NO'S 

M.HAEO  OF  OBSERVATIONS  (UX'CFN  SC  »  C  p  ) 
(NSAMP-NOrO)  VALUES  AF  L  TYPF  II  Cf.»S,i H" 

X 

m  m 

DATA  values 

AEPS 

m  m 

ABSOLUT!  CUUVERftE'-CF  "r 0 L t  &  a  n  C r  (J'.pi  T) 

ITEA 

m  m 

IOHRER  OF  ITERATIONS  P60U»r;  <RMT"»-  H") 

h  AX  1  T 

m  m 

PAXlVIt*  NU»  *  t  *  OF  ITERATION*  <  1  *:P  i  T  ) 

IMPLICIT  LiOUPL E  PRECISION  (A«n#  O-f) 
t  I  M  LESION  X  <11#  C  V  T  M  L  UFS)#  SHPTLI  ( I  *■ ) 


--  CCnrjCItuTS  CF  VAt'lATIC"  p0‘  “J"i'T  C  Tl  T  1  -  /  T  r  »• 
SHAPE  PA-A-r.  TE»» 

DATA  HTML  / *  p. / 


CATA 

CVTBL 

n.  # 

,'2?7?# 

.56T4S# 

.?i054. 

.?20C4, 

t 

.im?# 

,H*U?# 

.1hF3T# 

■  3 !  ?  i  0  # 

1 

.k«*a# 

.3012?# 

.OVJF*, 

.of.TJT# 

.  0  -  ?  r  r , 

1 

•  or  a  r. 

."mi# 

,  o  t  '  U  ?  > 

•  0  V  5  1  T  # 

J 

.041  FT# 

.BJISM# 

.m  2U 

•  o  1  /  4  4 , 

.M’TV 

r,  AT  A 

ShFT. l 

/!.#>. #3 

■  #  4  ,  ,  b  .  #  A 

■  #  F  ■  #  * ,  # 

» •  #  1 1  •  # 

:  l .  #  i  ? .  # 

% 

i  a  .  #  i  s ,  #  n .  #  i  t  . , 

U  ,#1«.# 

nr.,?'. 

#  4  *  .  #  S  '  . 

r  at  a 

cruc# 

CM  »  TCL* 

!'.  /"."H# 

i."-t#  : 

.’'..A/ 

—  stall  the  r *t a 

5CALf  ■  »U) 

Ot  If  !■!#  V  i  ? 

IF  I A  ll>  .ST,  S  C  ALE  >  SCAI  t  •  X  <  I  ) 

CCNT  Jf.Ul 

ru  ir.  i«n  vhs 

Xll)  ■  >  U>  /SCALE 
COST  1 Nn: 

IMS  •  AT  PE* 

XP  ■  AX * 

xp  ■  xr  /scale 

C  •  -LOO  (ONE  -P> 

IF  cr  PS  .EO.  Zf.tf)  >  EPS  •  T  "L**  1 ' 

—  CALCULATE  SUM/  SU“  OF  L'U'U  AM  S"  OF  S  u  U  A  <* :  '  "> 
SI  »  2T.mC 

l  ■  2  F  P  0 
SS  •  2FHC 
co  jc  i«i#  noi.s 

SI  •  SI  ♦  LOS  U  U)  /  X  p  ) 

S  «  t  ♦  x  II ) 

SS  •  SS  *  X  U )  *»  (  I) 

CCNT1XUL 
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C  —  USE  THF  C  OFF  F I C I  rNT  OF  VARIATION  TO  «T  «ETHOC-  OF 

C  HC“£NTS  ESTIMATE  OF  THE  SHAPr  PAFAMETEP 

CV  •  SOPT  (NOfcS  *SS  -S  *5)  /* 

IF  (CV  .SE.  CVT3L  <1>>  THE A' 

A  ■  SMPT8L  u> 

ELSE  IF  (CV  .IE.  CVVH  <«T«L>>  THE*1 
A  ■  SHPT3L  (HTbL) 

ELSE 

00  AO  1»2>  NT  *L 

IF  (CVTf  L  m  .LF .  cv>  THE* 
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MAKING  FISHER'S  EXACT  TEST  RELEVANT 
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Army  Materiel  Test  and  Evaluation  Directorate 
White  Sands  Missile  Range,  New  Mexico  88002-5175 


ABSTRACT 

The  Fisher-Irwln  Exact  Method  Is  made  relevant  by  Including  q-values  In 
the  analysis.  Q-values  are  post-test  Type  II  risks.  They  provide  Information 
which  complements  the  Type  I  risk  provided  by  the  p-value.  Reporting  both  the 
p-value  and  relevant  q-values  enables  managers  to  base  decisions  on  both  types 
of  risks.  For  references  on  q-values,  see  four  papers  by  Thrasher  In  the 
Proceedings  of  the  Thirtieth  through  Thirty-Third  Conferences  on  the  Design  of 
Experiments  In  the  Army  Research,  Development,  and  Testing,  U.S.  Army  Research 
Office,  4300  South  Miami  Boulevard,  Research  Triangle  Park,  North  Carolina 
27709-2211. 

Q-values  are  normally  calculated  using  the  same  algorithm  used  to  find 
pre-test  Type  II  errors.  The  q-value  calculation  Inputs  are  normally  (1)  the 
p-value  Instead  of  the  pre-test  Type  I  risk,'  (2)  the  sample  size  actually  used 
Instead  of  the  samole  size  planned,  and  (3)  the  same  relevant  values  of  the 
parameter  considered  In  the  pre-test  Type  II  risk  calculation.  Since  the 
Fisher-Irwln  Exact  Method  doesn't  historically  have  a  design  stage,  there  Is 
no  pre-test  algorithm  available  for  modification.  This  paper  develops  the 
necessary  algorithm  by  extending  the  p-value  calculation  based  on  the  binomial 
rather  than  the  hypergeometric  distribution. 

The  q-value  equations  are  developed  and  their  mathematical  properties  are 
examined.  Computer  programming  methods  are  discussed.  Examples  are  provided 
for  sample  sizes  both  (1)  small  enough  that  a  hand-held  calculator  can  be  used 
and  (2)  large  enough  to  require  a  digital  computer.  Numerical  results  are 
Interpreted  from  the  viewpoint  of  a  manager  that  must  balance  non-zero  Type  I 
and  Type  II  risks. 


INTRODUCTION  AND  OBJECTIVE 

The  Fisher-Irwln  Exact  Method  Is  a  quick  and  straightforward  technique  of 
comparing  two  samples  of  dichotomous  Items.  The  normal ly  reported  statistic 
from  this  test  Is  the  p-value.  The  p-value  is  the  probability  of  being  wrong 
in  marginally  rejecting  a  null  hypothesis  that  the  two  samples  are  from  one 
population.  In  practice,  managers  conclude  that  the  two  samples  are  from 
different  populations  if  they  believe  the  p-value  Is  sufficiently  low. 

This  method  of  analysis  has  not  gained  universal  acceptance.  The 
reluctance  to  use  this  method  may  well  be  due  to  unbalanced  reporting  of 
Information.  The  p-value  Is  used  to  report  the  Type-I  error.  This  error  Is 
sometimes  called  the  producer's  error,  the  contractor's  error,  or  the  error  of 
concern  for  the  advocates  of  maintaining  the  status  quo.  Traditionally  the 
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method  has  not  reported  Information  about  the  Type-1 1  error.  This  error  may 
be  called  consumer's  error,  the  government's  error,  or  the  error  of  concern 
for  the  advocates  of  change. 

The  Fisher-Irwin  Exact  Method  can  provide  relevant  information  about  the 
Type- 1 1  error.  This  additional  information  results  from  calculating  and 
reporting  q-values.  Q-values  are  the  probabilities  of  being  wrong  in 
marginally  falling  to  reject  the  null  hypothesis  Men  the  two  samples  are  from 
different  populations.  Since  the  two  populations  may  differ  in  different 
ways,  there  is  a  q-value  for  each  pair  of  separate  populations.  Managers  can 
use  a  q-value,  for  a  relevant  pair  of  unequal  populations,  as  evidence  for 
concluding  that  the  two  samples  are  from  those  different  populations.  They 
reach  this  conclusion  if  they  believe  a  relevant  q-value  is  sufficiently  high. 

This  paper  provides  equations  for  and  examples  of  calculating  (1)  the 
p-value  and  (2)  q-values  for  the  Fisher-Irwin  Exact  Method  using  a  one-sided 
analysis.  This  one-tailed  analysis  is  used  to  reject  a  single  population  in 
favor  of  two  populations  that  differ  in  the  direction  indicated  by  the  data. 

This  paper  also  discusses  a  digital  computer  program.  This  program  has 
been  written  to  (1)  handle  the  necessary  voluminous  calculations  for  large 
sample  sizes,  (2)  retain  the  analyst's  identification  of  the  two  measurement 
samples  and  the  two  mutually  exclusive  and  exhaustive  categories,  and  (3) 
provide  an  report  from  which  a  manager  can  decide  if  future  actions  should  be 
based  on  one  or  two  populations. 

The  Fisher-Irwin  Exact  Method  may  be  implemented  in  different  ways.  At 
the  cost  of  redundancy,  this  paper  uses  more  than  one  approach  to  illustrate 
different  viewpoints . 


P-VALUE  CALCULATION 

The  data  for  the  Fisher-Irwin  Exact  Method,  often  called  Fisher's  Exact 
Test,  consists  of  four  numbers.  They  and  their  sums  are  normally  arranged  In 
a  square  array.  The  following  array  has  double  entries  to  illustrate  both  the 
general  situation  and  a  specific  example: 


Category  1  =  Success : 

Category  2  *  Failure: 

Sum : 

Sample 

I  *  Development: 

r»19 

n-r=2 

n-21 

Sample 

II  *  Production: 

R-r-12 

(N-n.)-  (R-r  )*3  .. 

N-n«15 

Total  : 

- R75T - 

N-R»5 

Since  the  choices  of  Samples  I  and  II  and  of  Categories  1  and  2  are  both 
arbitrary,  there  are  four  possible  ways  the  data  can  be  arranqed.  The 
ambiguity  has  been  removed  from  the  above  table  by  naming  the  samples  and 
categories  to  make  (1)  n>(N-n)  and  (2)  r/n>(R-r) /(N-n) . 

There  are  two  methods  of  calculating  the  p-value.  The  best  known  uses 
the  hypergeometr 1c  distribution.  The  second  uses  the  binomial  distribution. 
Both  are  described  and  illustrated  in  pan*-.  1  95-203  of  Bradley,  James,  V., 
Distribution-Free  Statistical  Tests,  Pro  ;  it,.- Ha  11,  Inc.,  1968. 
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The  hypergeometric  approach  Is  based  on  a  population  of  N  Items  which  Is 
split  into  two  samples  of  sizes  n  and  N-n.  The  null,  hypothesis  Is  that  the 
difference  between  the  R  and  N-R  items  of  the  two  categories  did  not  Influence 
the  sample  selection.  The  probability  of  obtaining  the  data  is  the  ratio  of 
(1)  the  number  of  ways  items  from  one  category  can  be  chosen  for  the  two 
samples  to  (2)  the  number  of  ways  Items  of  this  category  can  be  chosen  from 
the  total  population.  Thus 


P [Obtain ing  the  Data] 


ncr  N-ncR-r 
NCR 


where  ^Cj  is  the  number  of  ways  of  choosing  j  items  from  1  Items.  Both  ^Cj 
and  iCi.j  are  found  from  the  following  relation  of  factorials: 


iCJ 


lCl‘J  "  J!  (1-j)l 


The  p-value  is  the  probability  of  obtaining  the  data  or  more  extreme 
partitions  of  the  N-R  items  of  Category  2.  (More  explicitly,  the  p-value  for 
the  one-sided  or  one-tailed  test  is  the  probability  that  the  partition  of  the 
N-R  items  will  be  as  unbalanced  as  the  data  in  the  direction  that  the  data 
suggests.)  For  the  specific  example,  the  p-value  is 


21 C2  15C3  21 C1  15C4  21C0  15C5 

p-value  - - +  . +  - . ^ . 

36C5  36C5  36C5 

21!  15!  21!  15!  21!  15! 

21  19!  31  121  _  11  20!  4!  11!  0!  21!  5!  10! 

36 !  +  361  +  361 

5!  311  5!  31!  5!  311 


=  .253  +  .076  +  .008  »  .34 


A  formal  expression  for  the  hypergeometr ic  approach  may  be  written  In  two 
ways.  Considering  the  possible  distribution  of  the  N-R  items  of  Category  2 
yields  one  of  the  two  following  equations: 
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If  (1)  n-r  Is  less  than  expected  because  n-r<n[(N-R)/N]  and  (2)  mj2  is 
the  minimum  possible  number  of  Items  In  Sample  I  from  Category  2, 

p-value  -  l  nC1  N.nCN.R.1  /  NCN_R; 


If  (1)  n-r  is  more  than  expected  because  n-r>n[(N-R)/N]  and  (2)  Mr«  Is  the 
maximum  possible  number  of  Items  In  Sample  I  from  Category  2,  " 


Mj2 

p-value  -  I  nC1  N.nCN^.i  /  ncn-R* 
1-n-r 


By  considering  the  distribution  of  the  R  Items  of  Category  1  Instead  of  the 
N-R  Items  of  Category  2,  this  formal  expression  is  written  using  two  other 
equations : 

If  (1)  r  Is  less  than  expected  because  r<n[R/N]  and  (2)  mji  Is  the  minimum 
possible  number  of  Items  In  Sample  I  from  Category  1, 


p-value  .  J  nC,  N.nCR_,  /  „CR; 


If  (1)  r  Is  more  than  expected  because  r>n[R/N]  and  (2)  Mn  Is  the  maximum 
possible  number  of  Items  In  Sample  I  from  Category  1, 


p-value  ■  J  nC,  /  NCR. 

1-r 


Binomial  Approach: 


The  binomial  approach  Is  based  on  an  infinite  population  from  which  two 
Independent  samples  are  taken.  Although  the  binomial  parameter  of  the 
distribution  of  Category  1  Items  may  be  estimated  by  R/N,  It  is  really 
unknown.  Fortunately  for  the  p-value  calculation,  this  parameter  will  cancel 
from  the  equations  regardless  of  Its  value.  Denoting  this  binomial  parameter 
by  p  allows  the  probabilities  of  obtaining  the  two  samples  to  be  written  as 


P[r;n,p  in  Sample  I]  ■  nCr  pr  (l-p)n'r  and 
P[R-r;N-n,p  In  Sample  II]  ■  ^.nCR.r  pR"r  (1-p) (N-n)-(R-r) 
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With  the  restriction  that  these  two  samples  are  independent,  the  probability 
of  obtaining  both  samples  is  the  product  of  the  two  above  equations;  this 
reduces  to 


P[0bta1ning  both  Samples]  n  nCr  N_nCR_p  PR  (l-p)N'R. 


The  probability  of  obtaining  one  big  sample  size  of  N  with  R  p-type 
observations  is 


P[R;N,p  in  Combined  Sample]  ■  NCR  pR  (l-p)N_R. 


Finally,  the  conditional  probability  of  obtaining  the  two  samples  given  that 
the  combined  sample  has  been  obtained  is 


P[(r  in  n)  and  (R-r  In  N-n)  |  (R  in  N)] 


P[Both  Samples] 
P[Comb1ned  Sample]’ 


This  equation  Is  expressed  In  terms  of  the  data  by  division  of  the  two 
previous  equations.  The  result  is 


ncr  N-n°R-r  PR  U'P)N'R  "Cr  N-ncR-r 

P [Obtain  1  ng  the  Data]  ■  . . .  . . 

NCR  p«(l-P)N-R  ncr 


This  is  the  same  equation  as  was  obtained  using  the  hypergeometr 1c  approach 
and  the  rest  of  the  calculation  of  the  p-value  proceeds  Identically. 

The  binomial  equations  In  the  above  paragraph  may  be  Illustrated  and 
clarified  by  using  (1)  the  data  from  this  discussion's  specific  example  and 
(2)  the  point  estimate  of  p  given  by  R/N«31/36".861.  The  result  Is 

P[19;21, .861  In  Sample  I]  -  21C19  ( .861 )19  (1-.861)21*19  -  .236, 
P[12;15,.861  In  Sample  II]  -  15C12  ( .861 )12  (1-.861)15’12  -  .203, 

P[0btain1ng  both  Samples]  ■  21cig  15^12  (1-.861  )36’3* 

■  .0479  ■  ( . 236 )( .203) , 
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P[31;36,.861  In  Combined  Sample]  ■  36C31  ( .861) 31  (1-.861)36*31 

■  .189,  and  by  dividing  equations 


PC (19  In  21)  and  (12  In  15)  |  (31  In  36)]  -  .0479/. 189  •  .253. 


The  value  of  .253  is  obviously  the  same  Intermediate  result  as  was  obtained  In 
the  hypergeometr 1c  approach.  In  fact,  any  value  of  p  yields  .253. 

It  Is  Illustrative  and  useful  to  obtain  P[31 ;36, . 861  In  Combined  Sample] 
without  the  assumption  that  all  36  Items  were  selected  from  one  population. 
This  Is  done  by  using  the  facts  that  U)  Sample  I  and  Sample  II  were  obtained 
Independently  and  (2)  the  31  items  In  Category  1  could  have  been  distributed 
between  the  two  samples  In  different  ways.  The  calculation  Is  summarized  In 
the  following  table.  The  starred  row  of  this  table  corresponds  to  the  data 
and  three  Intermediate  results  from  In  the  preceedlng  paragraph. 


R 

r' 

R-r' 

P[r';21,.861] 

P[R-r';15,,861] 

P[r 1 {21,.861]P[R-r 1 ;15, .861] 

31 

21 

10 

.0433 

.0348 

.00151 

31 

20 

11 

.147 

.0981 

.0144 

31 

19 

12 

.236 

.203 

.0479  * 

31 

18 

13 

.242 

.290 

.0700 

31 

17 

14 

.175 

.257 

.0450 

31 

16 

15 

.0961 

.106 

.0102 

TW" 

The 

last 

col umn 

contains  the 

probabilities  of 

the  different  ways  that  the  31 

Items  can  be  distributed.  The  sum  of  this  column  Is  the  probability  of  having 
31  Items  from  Category  1  In  the  two  samples.  The  value  of  .189  obviously 
agrees  with  the  shorter  calculation  in  the  preceedlng  paragraph.  A  modified 
version  of  the  longer  calculation  of  this  paragraph  will  be  needed  In  the 
calculation  of  q-values. 

Before  calculating  q-values,  it  Is  illustrative  and  useful  to  obtain  the 
p-value  from  the  table  of  the  preceedlng  paragraph.  Note  first  that  the  .0479 
In  the  last  column  of  the  starred  row  corresponding  to  the  data  agrees  with 
P[0bta1n1ng  both  Samples]  from  the  short  binomial  calculation  of  two 
paragraphs  ago.  Note  second  that  the  entries  above  .0479  In  the  table 
correspond  to  probabilities  of  obtaining  more  unlikely  partitions  than  the 
data.  Using  these  two  facts  yields 

p-value  «  P [Rej ect 1 ng  |  Rejection  Should  Not  Occur] 


.00151  j  .0144  i  .0479 
.189  .189  .189 


.34. 
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This  result  of  ,34  does  not  depend  on  the  number  used  for  p.  This  may  be 
seen  by  (1)  calculating  another  table  using  any  p  other  than  31/36*. 861  and 
(2)  summing  the  probabilities  of  obtaining  partitions  as  extreme  as  the  data. 

This  last  method  of  calculating  the  p-value  emphasizes  that  the  data  are 
viewed  marginally,  The  data  are  viewed  as  unbalanced  just  enough  to  warrant 
rejection  of  the  single-population  hypothesis. 

Q-VALUE  CALCULATION 

Q-values,  like  the  p-value,  consider  the  data  to  be  just  sufficiently 
unbalanced  to  warrent  rejection  of  the  single-population  hypothesis.  While 
the  p-value  is  the  probability  of  getting  results  at  least  as  unbalanced  as 
the  data,  q-values  are  the  probabilities  of  more  balanced  results. 

For  Fisher's  Exact  Test,  q-values  cannot  be  calculated  by  using  the 
hypergeometr 1c  approach.  All  q-values  are  conditional  probabilities  with  the 
condition  being  that  two  different  populations  provided  the  two  samples.  Thus 
q-values  must  be  calculated  by  using  the  binomial  approach  with  different 
binomial  parameters,  pr  and  pjr,  for  the  populations  of  the  two  samples.  If 
desired,  these  two  parameters  may  be  replaced  with  Pj  and  k  where  k*pj/pjj. 
Most  q-value  calculations  are  functions  of  only  one  parameter  and  lend 
themselves  to  a  two  dimensional  power  curve  representation,  For  the  Flsher- 
Irwln  Exact  Method,  there  are  two  parameters  so  the  representation  must  take 
the  form  of  a  three  dimensional  power  surface.  Any  specific  point  on  this 
surface  does  not  exhibit  as  much  Information  as  the  entire  surface.  To  be 
specific  In  the  following  calculation  however,  pt  and  pjj  will  be  taken  as  the 
point  estimates  from  the  data.  That  Is,  the  following  q-value  calculation 
will  address  the  error  of  concluding  that  p-pt-prt  when  actually 
pI*r/n*19/21*.905,  Pn*(R-r) /(N-n)*12/I5*.800,  and  k-pj/pn-, 905/, 800-1, 13. 
This  addresses  the  Intuitive  concern  of  "making  the  mistake  of  Ignoring  what 
the  data's  trying  to  tell  us." 

Since  a  q-value  for  any  specific  Pj  and  pj.  Is  the  probability  of  falsely 
retaining  the  assumption  that  one  p  describes  all  items,  a  q-value  Is  one 
minus  the  probability  of  rejecting  the  assumption  of  a  single  population  when 
there  are  two  populations  described  by  pj  and  prj .  This  can  be  calculated 
from  entries  In  a  table  of  probabilities  for  all  possible  values  of  r 
consistent  with  N,  n,  and  R. 

The  remainder  of  this  section  considers  two  approaches  to  the  q-value 
equation,  shows  that  the  two  results  are  equivalent,  and  discusses  some 
general  mathematical  properties. 
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For  the  specific  example  In  this  discussion,  one  table  used  to  calculate 
a  q-value  for  p j-19/21  and  pjj«12/15  Is 


R 

r' 

R-r' 

P[r ' ; 21 , .905] 

P[R-r '  ;15, . 800] 

P[r ' ;21, .9Q5]P[R-r 1 ;15, .800] 

31 

21 

10 

.122 

.103 

.0126 

31 

20 

11 

.270 

.188 

.0507 

31 

19 

12 

.284 

.250 

.0711 

31 

18 

13 

.190  ' 

.231 

.0438 

31 

17 

14 

.0898 

.132 

.0119 

31 

16 

15 

.0321 

.0352 

.00113 

7T9T 


The  sum  of  .191  on  the  lower  right  represents  the  probability  of  obtaining  a 
total  of  31  Items  from  Category  1  from  the  two  samples.  This  probability  can 
be  divided  Into  entries  In  the  rlaht  hand  column  to  find  the  conditional 
probabilities  of  obtaining  possible  numbers  of  Items  from  Category  1  In 
Samples  I  and  II.  Taking  the  data  and  more  extreme  divisions  of  the  R  Items 
from  category  1  as  evidence  of  rejection,  the  q-value  Is  found  from 


1  -  q-value  ■  P [Reject  1  ng  |  Rejection  Should  Occur] 


■  ‘°711  -OM7  .0126 

.191  +  .191  +  .191 


■  .70. 


Using  a  more  conventional  approach,  the  q-value  can  be  found  from  less  extreme 
divisions  of  the  R  Items  from  Category  1  to  be 


q-value  ■  P[Fa111ng  to  Reject (Rejection  Should  Occur] 


.0438  .0119  .00113 

.191  +  .191  +  .191 


■  .30. 
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This  procedure  may  be  stated  formally  with  the  following  equation: 


PO'jntPiMR-r'iN-n.Pn] 

r') 


max(r' ) 

I  PCr  •  {n.pjjPCR-r  •  ;N-n  ,pH] 
r'»min(r' ) 

where  mln(r')  and  max(r')  are  the  minimum  and  maximum  values  of  r'  allowed  by 
the  constraints  Imposed  by  fixed  values  of  N,  n,  and  R.  Increasing  of  r1  Is 
limited  by  the  total  size  of  both  Sample  I  and  Category  1.  That  Is,  r'  must 
simultaneously  satisfy  r'<n  and  r'<R.  Thus  the  upper  limit  on  the  above  sum 
is 


r-1 


n( 


max(r' )  ■  m1n(n,R) . 


Decreasing  of  r'  Is  limited  by  the  requirement  that  two  measurements  must  be 
non-negative.  Possible  measurements  of  Category  1  Items  In  Sample  I  and 
Category  2  Items  In  Sample  II  lead  to  r 1 >0  and  (N-n)-(R-r' )>0  «>'r'>n+R-N. 
Considering  the  other  two  possible  measurements  leads  to  four  other 
conditions:  R-r'<R,  R-r'<N-n,  n-r'<n,  and  n-r'<N-R.  These  four  conditions 
are  equivalent  to  the  first  two.  Thus  the  lower  limit  of  the  above  sum  Is 


m  1  n( r ' )  ■  max(0,n+R-N) . 


The  final  form  of  the  Category  1  equation  Is  thus 


q-val ue 


I  P[r'  ;n,pj  ]P[R-r  •  ;N-n  ,pH] 

r '■max(0,n+R-N) 


mln(n.R) 

I  P[r'  ;n,pj  ]P[R-r  *  ;N-n  ,pn] 

r'-max(0,n+R-N) 
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Using  Category  2  instead  of  Category  1  leads  to  the  following  table: 


N-R 

n-r  ‘ 

(N-n)-(R-r* ) 

P[n-r 1 ;21;2/21] 

P[(N-n)-(R-r* ) ; 1 5*  3/15 ] 

PCBoth] 

5 

0 

5 

.122 

.103 

.0126 

5 

1 

4 

.270 

.188 

.0507 

5 

2 

3 

.284 

.250 

.0711 

5 

3 

2 

.190 

.231 

.0438 

5 

4 

1 

.0898 

.132 

.0119 

5 

5 

0 

.0321 

.0352 

.00113 

7T5I 

The  numbers  in  the  right  three  columns  are  the  same  as  in  the  previous  table 
used  in  the  Category  1  approach.  The  calculation  proceeds  as  before  but  the 
indices  in  the  summation  appear  differently: 


1  -  q-value 


f  P[n-r ' }21,,0952]P[ (N-n)-(R-r' );15, .200] 
n-r '-0 


J  P[n-r 1 ;21, .0952]P[ (N-n)-(R-r' );15..200] 
n-r'*0 


.0126  ,  .0507  .0731 

.191  +  ”“.191  +  .191 


.70  or 


{  P[n-r 1 ;21, .0952]P[ (N-n)-(R-r 1 );15, .200] 
n-r' "3 

q-value  ■  ■■■■■— . . . . . ■■■ ■- . — . . , 

l  P[n-r ' ;21, .0952 ]P[ (N-n)-(R-r' ) ;15, .200] 
n-r ' «0 


.0438  .0119  _  .00113 

.191  +  ".191  +  .191 


.30. 


This  procedure  may  be  stated  formally  with  the  following  equation: 


q-val ue 


max(n-r' ) 

7  PCn-r';n,l-PI]PC(N-n)-(R-r');N-n,l-pII] 

n-r  '«n-r+l 

max(n-r' ) 

l  P[ n-r '  ;n,l-pj]P[ (N-n)-(R-r 1 ) jN-n^l-Pj  j] 

min(n-r' ) 
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where  mln(n-r')  and  max(n-r')  are  the  minimum  and  maximum  possible 
measurements  of  Category  2  Items  In  Sample  I.  By  arguments  similar  to  those 
used  In  the  Category  1  approach,  n-r'<n  and  n-r'<N-R  Irppl y  that 


max(n-r' )  *  m1n(n,N-R) 

and  n-r>0  and  R-r ' >0  ^  n-r'>n-R  imply  that 
min(n-r' )  ■  max(0,n-R) . 


The  use  and  Interpretation  of  this  result  must  of  course  be  done  In 
conjunction  with  n>(N-n)  and  r/n>(R-r)/(N-n) .  Interpretation  must  recognize 
that  the  arrangement  of  the  data  in  a  standard  format  may  or  may  not  select 
Category  1  as  the  category  of  primary  Interest  and/or  Sample  I  as  the  first 
sample  drawn  and/or  tested. 

Equivalency  of  Methods; 

Although  the  equations  from  the  Category  1  and  Category  2  approaches 
appear  quite  different,  they  are  equivalent.  By  using  the  binomial 
probability  relation  P[1 ;m,p]"P[m-1 ;m,l-p] ,  the  Category  2  equation  may  be 
rewritten  as 


mln(n.N-R) 

l  P[>*'  ;n »Pt ]P[R-r '  ;N-n .Pit] 

n-rWr+1 

q-value-  tonh - 

l  PCr'jn.pjHR-r'jN-n.Pn] 

n-r '-max(0,n-R) 

The  limits  on  the  summations  may  be  rewritten  by  using  m1n(m,M)«-max(-m,-M) , 
k+max(m,M)-max(  k+m,k+M) ,  and  k+m1n(m,M)«m1n(  k+m,k+M)  to  yield 

max(0,n+R-N) 

l  PCr'jn.pjlPCR-r'jN-n.pjj] 

*  I  m  _  1 


max(0,n*R-N) 

l  PCr'jn.p^PCR-r'jN-n.pjj] 
r'»min(n,R) 


By  reversing  the  summation  limits  to  correspond  to  the  normal  practice  of 
summing  from  low  to  high  Indices,  this  equation  becomes  the  Category  1  result. 
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Range,  Sum  with  P-Value,  and  Symmetr; 


The  q-value,  like  any  other  probability,  Is  bounded  by  zero  and  one. 
This  Is  verified  for  the  Category  1  equation  by  splitting  the  sum  In  the 
denominator  Into  two  sums  which  first  sum  from  r'«m1n(r' )«max(0,n+R-N)  to 
r'-r-l  and  then  summing  from  r'»r  to  r 1  »ma>c( r *  )«»m1n(n,R).  Dividing  both 
numerator  and  denominator  by  the  sum  In  the  numerator  then  yields 


q-value 


m1n(n,R) 

l  PO'in.p^PCR-r'jN-n.pjj] 


r'«max(C,n+R-N) 


P[r'  .n.PiJPCR-r'.N-n.pn], 


Since  this  equation's  ratio  of  the  two  sums  Is  never  negative  but  It  will  be 
Infinite  If  the  data  yields  r«max(0,n+R-N),  q-value  >  0.  Since  this  ratio  can 
be  0  by  choosing  pj  and  pjj  equal  to  0  or  1,  q-value  <  1. 

When  P|  equals  pjj,  the  q-value  Is  one  minus  the  p-value.  This  occurs 
because  the  possible  values  of  r'  are  divided  Into  two  mutually  exclusive  and 
exhaustive  sets.  One  set  contains  possible  measurements  as  unlikely  or  more 
unlikely  than  r.  The  other  contains  values  of  r'  more  likely  than  r,  These 
two  sets  Identify  conditional  probabilities  that  are  summed  to  find  the 
p-value  and  q-value.  The  p-value  summation  uses  the  unlikely  set  with  both  pr 
and  p t r  equated  to  any  common  probability.  The  q-value  summation  uses  the 
likely  set  with  any  values  of  p»  and  pj r .  The  mutual  exclusiveness  and 
exhaustiveness  of  the  two  sets  require  that  p-value  +  q-value  ■  1  when  pj*pjj, 

For  the  Flsher-Irwln  Exact  Method,  the  q-value  Is  symmetric  about  the  off 
diagonal  In  a  plot  of  pj  versus  pj j .  That  Is,  symmetry  Is  expressed  by 

q-value(N,n,R,r .pj.pjj)  ■  q-val ue(N,n,R,r ,l-pjj,l-pj) . 


This  may  be  seen  by  applying  the  binomial  equation  P[1;m,p]  -  mC^  p1  (l-p)r 
to  the  Category  1  equation  In  a  series  of  three  equations: 


q-value 


r '»max(0,n+R-N) 


P[r* ;n ,pj ]P[R-r ' ;N-n ,pjj] 


m1n(n,R)  „ 

l  PLr'jn.p^PCR-r'.N-n.pn] 

r'«max(0,n+R -N) 
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r-l 


icr'  Pf  U-Pi) n-r  N-ncR-r '  PlI 


r '«max(0,n+R-N) 


R-r1 


m1n(n,R)  ,  , 

.1  n  .  ncr'  Pir  (1-Pi)n“r  N-ncR-r'  Pi  I 
r '»max(0,n+R-N) 


R-r 


(No  r'  F»ctor)  rf  „Cr,  N_„CR_r,  p,r'  (l-p,>-r' 
_ r^max^n+R-N) _ 


(No  r*  Factor)  rjl  nCr.  N_nCR_ri  pjr '  (1  *px) ”r‘ 
r '■max(0,n+R-N) 


(l.PlI)N-n-R+r' 

(i-Pn)N'n“R+r' 

p 1 1 _r '  (i-Pn)r’ 

Pn’fl  (i-pi i)r‘ 


where  the  (No  r'  Factor)  Is  (l-pj)n  PjjR  (1-p? r)^"n"R.  This  constant  has  been 
factored  from  each  term  of  the  sum  over  r*  .  After  canceling  this  factor  from 
the  equation,  the  symmetry  Is  evident  because  substituting  1-pn  for  pr  and 
1-Pj  for  pji  yields  the  same, equation , 

The  q-value  for  the  Flsher-Irwln  Exact  Method  also  Is  symmetric  In  n  and 
R.  Applying  *C<  m  1l/Jl(1-J)l  to  the  above  equation  and  canceling  nl  and 
(N-n)l  yields  J 

^  * _  f Pi  (l-Pn)l  r 

r'.m«*(6,n+R-N)  r' 1  (n’r'>1  <R*r'i 1  (N-n-R+r‘)l  (1-p,)  pnJ 

q- value  ■  . - — .  ■■  ■  ■  .  - — 

mln^n.R)  1  fPj  (l*Pn)"|r' 

r'-max(S.n+R-N)  •*' 1  (R"r')1  (N-n-R+r* ) l  [UT{)  Pn  J 


Substitution  of  n  for  R  and  R  for  n  yields  the  same  equation.  Thus  symmetry 
Is  expressed  by 


q-value(N,n,R,r,pI,pII)  -  q-value(N,R,n ,r .ppPjj) . 


This  equation  reflects  the  mathematical  arbitrariness  in  identifying  samples 
and  categories.  The  samples  and  categories  normally  are  distinguished 
physical ly;  but  they  are  Interchangeable  mathematically. 
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RECAPITULATION  AND  INTERPRETATION 

The  p-value  and  a  relevant  q-value  can  provide  influencing  factors  for 
management.  If  the  p-value  is  lower  than  the  risk  allowed  for  the  proponent 
of  a  single  population,  management  Is  Inclined  toward  the  decision  that  two 
populations  exist.  If  a  relevant  q-value  Is  higher  than  the  risk  that  the 
proponent  of  two  populations  is  willing  to  take,  management  Is  also  Inclined 
toward  the  decision  that  two  relevant  populations  exist.  On  the  other  hand,  a 
high  p-value  or  low  relevant  q-value  Inclines  management  toward  the  decision 
that  there  is  one  population. 

Management  will  quite  often  be  Influenced  by  factors  other  than  the 
p-value  and  a  relevant  q-value.  A  subjective  decision-making  process  will 
naturally  be  used  to  consider  all  factors.  The  extremity  of  the  lowness  or 
highness  of  the  p-value  and  a  q-value  provides  the  subjective  weight  for  these 
two  factors . 

If  management  cannot  determine  threshold  risks  to  Indicate  two 
populations  when  the  p-value  Is  below  and  a  q-value  Is  above  these  thresholds, 
an  alternate  approach  Is  to  compare  the  p-value  and  a  q-value.  Management  can 
set  a  threshold  ratio  of  Type  II  to  Type  I  risks  and  compare  a  ratio  of 
q-value/ p-value  to  this  threshold.  Two  populations  are  then  Indicated  If  a 
ratio  of  q-val ue/p-val ue  Is  too  high.  In  a  subjective  decision-making  process 
considering  many  factors,  the  extremity  of  a  q-val  ue/p-val  ue  ratio  provides 
the  subjective  weight  of  the  Fisher's  Exact  Test  factor. 

Management  should  determine  which  two  populations  are  relevant.  Factors 
other  than  the  data  may  suggest  specific  populations.  Management  should 
consider  a  q-value  for  each  and  every  pair  of  relevant  populations.  If  the 
analyst  Is  not  provided  with  the  pj  and  Pn  for  any  relevantly  different 
populations,  the  report  to  management  should  include  a  table  of  q-val ues  for  a 
wide  range  of  pj  and  pjj. 

For  the  primary  example  In  this  discussion,  .34  Is  the  p-value  and  .30  is 
a  q-value  for  the  two  populations  suggested  by  the  data.  F  these  two 
populations  with  pj -.9  and  p j j -.8  are  relevantly  different,  the  two  risks  of 
. 34  and  .30  provide  the  basis  for  action.  If  the  existence  of  these  two 
populations  Is  considered  as  positive,  .34  is  the  probability  of  making  a 
false  positive  decision.  Similarly,  considering  the  existence  of  only  one 
population  as  negative  Implies  that  .30  is  the  probability  of  making  a  false 
negative  decision . 

If  .34  and  .30  are  believed  sufficiently  low  and  high  for  probabilities 
of  false  positives  and  negatives  respectively,  future  action  Is  based  on  the 
existence  of  two  populations  with  with  pj  and  p r j  estimated  by  .9  and  .8.  If 
.34  and  .30  are  believed  sufficiently  high  and  low,  future  action  Is  based  on 
a  single  population . 
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For  this  example,  .30/.34»l/l.la.9  is  a  ratio  of  q-val ue/ p-val ue . 

Subject  to  the  relevancy  of  pr«.9  and  Ptt*.8,  1/1. 1«. 9  is  the  ratio  of  risks 
of  making  false  negative  and  false  positive  decisions.  Future  action  Is  based 
on  two  populations  If  1/1.1-.9  Is  believed  sufficiently  high.  Similarly, 
future  action  is  based  a  single  population  If  1/1. 1». 9  Is  believed 
sufficiently  low. 

If  the  p-val ue  and  a  q-val ue  provide  conflicting  or  Indeterminate 
Indications  that  are  unresolvable,  the  immediate  future  action  Is  to  do 
additional  testing.  Additional  testing  should  provide  more  definitive 
Information  by  yielding  either  a  low  p-val ue  and  a  high  q-val ue  or  a  high 
p-value  and  a  low  q-value.  Naturally  increasing  the  sample  sizes  may  not 
yield  a  proportional  Increase  In  all  the  data;  but  If  additional  testing 
actually  doubled  all  the  data  In  this  paper's  example,  the  results  would  be 
.18  for  the  p-value,  .35  for  a  q-value,  and  .35/ .18*2  for  a  q-val ue/p-value 
ratio  corresponding  to  p j- . 9  and  Pn*.8.  This  possible  decrease  In  the 
p-value,  Increase  In  a  q-value,  and  Increase  in  a  ratio  of  q-val ue/ p-val ue 
would  Increase  the  tendency  to  base  future  actions  on  two  populations. 

COMPUTING  METHODS  AND  RESULTS 

A  digital  computer  program  has  been  written  In  Pascal/3000  to  facilitate 
the  p-value  and  q-value  analysis  of  the  Fisher-Irwln  Exact  Method. 

Two  related  manipulations  are  useful  In  extending  the  range  of  data  which 
yields  q-val ues  without  computer  overflows  or  underflows.  The  equation  for 
the  q-value  can  be  rewritten  as 


c 

pI 

r' 

r'«max(o,n+R-N)  '"1  I""")'  (*"•')•  (N-n-R+r')l 

_(l*Pl)  Pi i_ 

m1n(n,R)  C 

’Pi  (l-Pii)" 

r ' 

]  v  r'l  (n-r’)i  (R-r')l  (N-n-R+r1)! 

r '■max(0,n+R-N) 

_(!-p i)  Pi i. 

where  C  Is  any  constant.  The  computer  program  can  assign  C  with  a  value  which 
hinders  the  summed  terms  from  exceeding  the  computer's  working  range.  To  make 
this  assignment  without  overflowing  or  underflowing  the  computer,  each  term 
must  be  considered  as 


_ C _ 

r '  1  ( n-r ' ) !  (R-r1 ) I  (N-n-R+r' ) I 


pi  (1-pii) 
(  i  ■■  p  i  )  p  1 1 


r ' 


ex  p[  In  (TERMS)] 
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nere  the  expression  1  n( Tf£RMS)  in  the  exponential  is 
In (TERMS)  *  ln[C] 

+  r'Cln(pj)  +  ln(l-pn)  -  lrt(l-pI)  -  ln(pn)] 

-  ln[r't]  -  ln[(n-r')l]  -  ln[(R-r')!]  -  ln[(N-n-R+r' )  I  ]. 


The  constant  C  can  be  selected  to  keep  ln(TERMS)  within  the  computer's  range 
for  x  in  exp(x).  (e.g.  -176  to  176).  For  any  value  of  r' ,  this  selection  can 
then  be  used  to  force  C  exp( 1 n ( TERMS ) )  Into  the  computer's  operating  range 
(e.g.,  8.6(10)“7a  to  1.15  (10)77).  Naturally  this  programming  technique  is 
successful  only  if  r'  doesn't  change  too  much  in  the  summation  between 
max(0,n+R-N)  and  min(n,R). 

The  range  of  computer  calculations  for  the  p-value  can  be  extended  by 
using  logarithms.  One  useful  form  of  the  p-value  equation  is 


p-valpe  ’*  l  exp[ln(nCx)  +  ln(N_nC„)  -  ln(NCz)] 
1»v 


where  the  factors  v,  w,  x,  y,  and  z  are  dependent  on  r,  R,  and  N  according  to 
the  following  table: 


If  IK(N-R) 


If  R>(N-R) 


Factor  If  R<n  if  R>n 


if  (N-RX(N-n)  If  (N-R)s(N-n) 


v 

w 

X 

y 

z 


r  r 

R  n 

1  i 

R-1  h-1 

R  R 


'N-n)-(k-r) 


U- R 
N-R-n 

i 

N-R 


(N-n)-(R-r) 
N-n 
N-R- 1 
i 

N-R 


The  computer  program  operates  from  a  terminal .  At  the  start  of  the 
program,  the  user  selects  either  the  terminal  screen  or  a  printer  for  the 
program  output.  The  Information  in  Figure  1  then  appears  on  the  screen.  This 
provides  the  user  with  a  brief  summary  of  the  analysis  and  asks  the  user  which 
four  Independent  variables  will  be  entered.  Figure  'l  provides  an  example  of 
the  terminal  screen  after  the  output  has  been  directed  to  a  printer  and  the 
user  has  chosen  to  enter  N,  R,  and  r.  Figures  3a  and  3b  contain  the  output 
for  this  input.  Correspondingly,  Figures  4  and  5  show  input  and  output  jhen 
the  user  has  entered  r,  n-r,  R-r,  and  (N-n)-(R-r).  Finally  Figure  6  shows  an 
example  with  both  input  and  output  on  the  terminal  screen.  For  this  example, 
the  input  produces  a  standard  table  in  a  different  order  than  the  data. 
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Seme  results  from  the  outputs  in  these  figures  (and  similar  computer 
executions)  are  compiled  In  the  following  table.  All  possible  measurements  of 
r  for  N*36,  n*21,  and  R*31  are  Included.  The  tabulated  q-values  are 
referenced  to  e  and  4,  instead  of  pr  and  pjj.  This  is  necessary  because  the 
computer  places  the  data  in  a  standardly  ordered  table  sometimes  making  p j -e 
and  pjj -4,  and  sometimes  resulting  in  pj*$  and  Pjj*e. 


point-estimates  q-value  for 

to  replace  .861  — - -  ■  — — — . — 


£ 

p-value 

£ 

£ 

point 

estimates 

9*. 8 

A*  .9 

9*.  85 
a*. 87 

6*.  87 
^*!  .85 

9*.9 

fa*. 8 

16 

.054 

.762 

1.000 

.000 

.791 

.925 

.962 

.993 

17 

.292 

.810 

.933 

.249 

.381 

.644 

.766 

.924 

18 

.663 

.857 

.867 

.306 

.098 

.274 

.407 

.682 

19 

.337 

.905 

.800 

.297 

.902 

.726 

.593 

.318 

20 

.084 

.952 

.733 

.241 

.988 

.941 

.884 

.690 

21 

.008 

1.000 

.567 

.000 

.999 

.995 

.987 

.941 

This  table  emphasizes  that  management  needs  to  determine  relevantly  different 
populations  Instead  of  just  considering  the  point-estimates  suggested  by  the 
data.  As  expected,  the  extreme  r  measurements  of  16  and  21  lead  to  low 
p-values  Indicating  two  populations.  The  two  populations  Indicated  however, 
are  not  those  suggested  by  the  point-estimates  of  9  and  4,.  (9  and  4,  are  the 
binomial  probabilities  that  describe  the  two  populations;  they  replace  the 
common-population  point-estimate  of  31/36*. 861.)  The  q-values  for  those 
point-estimates  are  Identically  zero.  Although  every  manager  Is  free  to 
determine  how  high  a  q-value  needs  to  be  for  a  two  population  decision,  these 
are  low  by  any  standard.  Management  must  realize  that  no  amount  of  testing 
can  prove  that  anything  Is  either  completely  perfect  or  worthless.  Instead, 
more  reasonable  values  of  e  and  4,  must  be  considered.  If  e».8  and  a*. 9  are 
considered  for  r«l6  (or  e».9  and  4*. 8  for  r*21),  the  q-value  of  .791  (or  .941) 
Is  quite  high.  Even  higher  q-values  are  obtained  when  e-.85  and  4,- . 87  are 
considered  for  r»16  (or  9*. 87  and  $*.85  for  r«21).  This  reflects  the  fact 
that  It's  easier  to  say  two  things  are  different  If  they  don't  have  to  be  very 
different.  Considering  possible  populations  for  which  9  and  4,  are  on  opposite 
sides  of  the  common- po pul  at  Ion  point-estimate  of  31/36*. 861  from  the  point- 
estimates  (e.g.  considering  e«.9  and  *-.8  for  r*16  or  9*. 8  and  4,-. 9  for  r*21) 
lead*  to  very  high  q-values.  This  reflects  the  compound  fact  that  (1) 
obtaining  data  biased  In  opposite  direction  from  two  existing  populations  Is 
extremely  unlikely  so  (2)  the  existence  of  such  data  strongly  Implies  more 
than  one  population.  Less  extreme  but  similar  results  are  obtained  for  r 
measurements  of  17  and  20.  An  r  measurement  of  18  Indicates  one  population 
unless  management  is  concerned  about  very  extreme  alternate  populations. 
Finally  an  r  measurement  of  19  Indicates  two  populations  If  management  Is 
careful  about  what  those  two  populations  are. 
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The  dependence  of  both  the  p-value  and  q-val ue  on  the  number  of 
measurements  Is  Illustrated  by  the  following  example.  Four  measurements  are 
assumed  to  yield  values  of  N,  n,  R,  and  r  given  by  {N1,N2,N3,N  } 

{20,40,80,160},  {n1,n(  . .  “  “  ‘ 

and  (VVVr ,} 

are  just  multiples  of  the  first.  All  four  of  these  hypothetical  measurements 
provide  point  estimates  of  $  and  $  of  9/10-.9  and  ( 1 7-9 ) /10». 8  respectively. 


'YW  “  {lu»20*40,80},  -V, 34, 68, 136}, 

{§, 18, 36,72}.  The  second,  third  and  fourth  measurements 


The  Interior  of  the  following  table  contains  sets  of  q-values  from  four 
executions  of  the  computer  program.  Each  set  has  the  q-value  for  the  smallest 
sample  size  first  and  the  largest  last. 


e\d> 

.76 

.80 

.84 

.86 

.90 

.94 

.279,. 365,. 447,. 5261 
.180, .217, .232, .2211 
.034, .082, .059, .027( 

\  \ 

.352, .472,. 595, .721 
.237, .303, .356, .397' 
.118,. 127,. 111,. 074’ 

r 

.444,  .599,. 752,  ,885 
.316, .421, .525, .633 
.169,. 201,. 209,. 192 

The  corresponding  set  of  p-values  Is  1.500, .331, .174, .060}.  Note  (A)  that 
Increasing  the  sample  size  decreases  the  p-value  and  Increases  the  q-value  for 
e«.9  and  $«.8.  Thus  Increasing  the  sample  size,  If  the  data  remains 
proportionate,  Increases  the  justification  for  deciding  that  two  population 
yielded  the  two  samples.  Note  also  (B)  that  the  p-value  has  a  more  pronounced 
change  than  the  q-values.  Thus  the  prvalue  Is  more  sensitive  than  the 
q-values.  A  unusually  high  q-val  u£  thus  has  at.  least  as  much  significance  as 
an  unusually  low  p-value.  Note  finally  (C)  that  Increasing  the  sample  size 
when  e*.90  &  $*.76,  9*.94  &  $">.76,  e“» 94  &  $*,80,  and  9*. 94  &  $».84  eventually 
leads  to  a  decrease  In  the  q-value.  This  corresponds  to  universal  measurement 
Implying  exact  results,  (This  large-measurement  effect  does  not  occur  for 
9*. 86  &  $*.76,  e*. 86  &  $■ . 80 ,  0«. 86  &  $».84,  and  e-,90  &  $“,84  because  they 
are  on  the  opposite  side  of  the  point  estimate  from  the  extreme  Identified  by 
the  alternate  hypothesis.) 


The  program  Is  designed  so  the  user  can  keep  track  of  a  sample  of 
prominence  and  a  category  of  Interest,  This  enables  the  user  to  enter  and 
analyze  management's  relevantly  different  populations.  For  example,  consider 
the  hypothetical  case  analyzed  In  Figures  7a  and  7b,  Suppose  that  a  field- 
fired  missile  Is  being  developed.  Enough  tests  have  been  made  on  the  Initial 
design  to  obtain  107  hits  and  14  misses.  A  set  of  shorter  missile  fins  In 
proposed  to  make  the  field-assembly  faster.  A  short  series  of  tests  on  the 
short-fin  version  yields  11  hits  and  3  misses.  The  short-fin  test  Is 
prominent  In  the  mind  of  the  missile  designer;  the  short  fins  should  not  be 
used  If  they  significantly  degrade  the  missile's  performance.  Figures  7a  and 
7b  contain  the  Input  and  terminal-screen  output  of  an  analysis  using  the 
Flsher-Irwln  Exact  Method.  The  first  entry  Into  the  computer,  11,  Identifies 
the  short-fin  test  as  the  sample  of  prominence  and  hits  as  the  category  of 
interest.  The  p-value,  .249,  Is  somewhat  low  but  the  advocates  of  fast 
assembly  with  short  fins  might  claim  that.  .249  is  not  close  enough  to  zero  to 
warrant  the  conclusion  that  short  fins  have  degraded  the  missile's  accuracy. 
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The  q-value  for  $=11/14=. 786  and  9=107/121=.  884  is  .375.  That  is  slightly 
higher  than  the  p-value  but  It  might  not  be  large  enough  justify  not  using  the 
short  fins.  If  management  sets  the  desired  requirement  at  q«,900  and  decrees 
that  $=.850  Is  an  unacceptable  accuracy  rate,  the  last  table  on  Figure  7b 
provides  a  basis  for  decision.  The  q-value  for  q-,900  and  $».850  Is  .525. 
Since  this  Is  twice  the  p-value,  management  has  a  fairly  strong  basis  for  not 
using  the  short  fins.  If  management  leaves  the  desired  requirement  at  .900 
and  raises  the  unacceptable  level  to  .890,  the  q-value  increases  to  .706.  The 
argument  for  rejecting  the  short  fins  Is  thus  quite  strong  If  .890  is  really 
an  unacceptable  accuracy  rate. 


SUMMARY 

The  p-value  and  q-value  analysis  of  the  Fisher-Irwln  Exact  Method  has 
been  developed.  The  p-value  equation  has  been  derived  using  two  techniques; 
hypergeometric  and  binomial.  The  binomial  technique  has  been  extended  to 
yield  a  q-value  equation.  This  equation  has  been  derived  from  two  sources: 
possible  category  one  measurements  and  possible  category  two  measurements. 

This  q-value  equation  has  been  shown  to  possess  mathematical  symmetry.  The 
q-value  for  Pi*Pn  has  been  shown  to  equal  one  minus  the  p-value;  this  was 
predestined  for  the  Fisher-Irwln  Exact  Method  because  It  Is  a  general  property 
of  the  p-value  and  q-values.  A  computer  program  has  been  written.  This  makes 
the  analysis  practical.  Analysts  can  perform  voluminous  calculations  without 
approximations.  Managers  can  consider  the  relative  sizes  and  Importance  of 
the  p-value  and  relevant  q-values.  Managers  can  lecide  If  the  two  samples  are 
from  one  population  or  from  two  populations  differing  either  (1)  from  the 
combined  point  estimate  of  the  population  or  (2)  according  to  (A)  a  desired 
population  or  standard  and  (B)  an  unacceptable  population.  Computer  generated 
reports  have  been  provided  for  communication  between  analysts  and  managers. 

The  development  of  the  p-value  and  q-value  analysis  of  the  Flsher-Irwin  Exact 
Method  has  reached  the  stage  of  Implementation. 

CONCLUSION 

The  analyst  has  a  responsibility  to  report  all  Information  influencing 
the  decision.  This  information  should  be  in  a  form  that  can  be  understood  and 
used  by  the  decision-maker.  Reporting  the  p-value  and  relevant  q-values 
satisfies  both  of  these  conditions.  The  p-value  and  q-values  provide  the 
decision-maker  with  estimates  of  the  risks  of  making  wrong  decisions.  This 
makes  Fisher's  Exact  Test  relevant. 
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Greetings!  Welcome  to  a  computerized  Fisher-Irwln  Exact  Test  Analysis. 


Two  Independent  samples  are  Initially  assumed  to  be  from  a  single  population. 
This  assumption  is  rejected  and  the  two  samples  are  considered  to  represent 
two  statistically  different  populations  if  management  reaches  two  conclusions: 

1)  The  p-value  is  deemed  sufficiently  low  and 

2)  A  q-value  for  relevantly  different  populations  is  deemed  sufficiently  high. 

The  q-value  is  the  probability  of  falsely  deciding  that  two  populations  exist. 
A  q-value  for  two  relevantly  different  populations  1.s  the  probability  of 
falsely  deciding  that  those  two  populations  are  one  population. 

This  computerized  analysis  does  a  one  sided  test  In  the  direction  Indicated  by 
the  data.  It  requires  four  numerical  inputs  determining  nine  numbers: 

Category  One  (e.g.  Success):  Category  Two  (e.g.  Failure):  Sum: 
Sample  One:  r  n-r  n 

Sample  Two:  R-r  (N-n)-(R-r)  N-n 

Total :  R  N-R  N 

Data  may  be  entered  In  two  ways.  The  theoretical-statistician  approach  uses 
"N,  n,  R,  &  r" .  The  reliability-engineer  uses  "r,  n-r,  R-r,  ?>  (N-n)-R-r)" . 

ENTER  RELIABILITY-ENGINEER/ THEORETICAL-STATISTICIAN  APPROACH  "R/S" 


Figure  1.  Terminal  screen  at  program  Initiation. 


350 


ENTER  RELIABILITY-ENGINEER/THEORETICAL-STATISTICIAN  APPROACH  "R/S"  s 


ENTER  SIZE  OF  POPULATION  "N"  36 

ENTER  SIZE  OF  SAMPLE  OF  PROMINENCE  "n"  21 

ENTER  #  OF  ITEMS  FROM  CATEGORY  OF  INTEREST  IN  POPULATION  "R"  31 

ENTER  #  OF  ITEMS  FROM  CATEGORY  OF  INTEREST  IN  SAMPLE  OF  PROMINENCE  "r"  19 

ENTER  "T"  FOR  TABLE  OF  Q-VALUES,  "ANYTHING  ELSE"  TO  SKIP  TABLE  t 

ENTER  "C"  FOR  CLOSE  LOOK  AT  Q-VALUE  TABLE  IN  DATA  SUGGESTED  REGION, 
"ANYTHING  ELSE"  TO  SKIP  c 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS' TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  ( I.E.  REPLACE  e  ■  19  /  21  -  0.905)  AND  THE 
SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  ■  12  /  15  ■  0.800), 

"ANYTHING  ELSE"  TO  SKIP  m 

ENTER  "0"  .9 

ENTER  V  .8 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  9  •  19  /  21  ■  0.905)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  ■  12  /  15  ■  0.800), 

ANYTHING  ELSE"  TO  SKIP  in 

ENTER  "0"  .87 

ENTER  V  .85 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  0  ■  19  /  21  -  0.905)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  ■  12  /  15  ■  0.800), 

"ANYTHING  ELSE"  TO  SKIP  tn 

ENTER  "0"  1 

ENTER  V  .7 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  0  -  19  /  21  ■  0.905)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  ■  12  /  15  -  0.800), 

"ANYTHING  ELSE"  TO  SKIP  skip 

END  OF  PROGRAM 


Figure  2.  Sample  of  theoretical-statistician  input. 
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ANALYSIS  OF  FISHER'S  EXACT  TEST 


In  the  following  standardly  ordered  table,  the  sample  of  prominence  and 
category  of  Interest  are  Identified  by  the  user  as  Sample  I  and  Category  1, 

Category  1:  Category  2: 

Sample  I:  r  -  19  n  -  r  ■  2  n  -  21 

Sample  II:  R  -  r  ■  12  (N  -n)  -  (R  -  r)  ■  3  N  -  n  ■  15 

R  -  31  N  -  R  ■  5  N  ■  36 

For  this  data,  the  post-test  risk  of  a  Type  I  error  Is  p-value  ■  0.337. 

For  this  data's  two  suggested  binomial  parameters  of  the  category  of  Interest 
(l.e.  9  *  19  /  21  ■  0.905  and  a  ■  12  /  15  «■  0.800),  the  post-test  risk 
of  a  Type  II  error  Is  q-value  ■  0.297. 

For  other  binomial  parameters  of  the  category  of  Interest,  q-values  may  be 
estimated  from  the  following  table: 


e  \  4 

0.050 

0.150 

0.250 

0.350 

0.450 

0.550 

0.650 

0.750 

0.850 

0.950 

0.050 

0.663 

0.956 

0.990 

0.997 

0.999 

1.000 

1.000 

1.000 

1.000 

1.000 

0.150 

0.182 

0.663 

0.866 

0.946 

0.978 

0.992 

0.997 

0.999 

1.000 

1.000 

0.250 

0.058 

0.389 

0.663 

0.827 

0.917 

0.963 

0.986 

0.996 

0.999 

1.000 

0.350 

0.020 

0.210 

0.456 

0.663 

0.809 

0.903 

0.958 

0.986 

0.997 

1.000 

0.450 

0.007 

0.105 

0.285 

0,483 

0.663 

0.804 

0.903 

0.963 

0.992 

1.000 

0.550 

0.003 

0.048 

0.158 

0.314 

0.491 

0.663 

0.809 

0.917 

0.978 

0.999 

0.650 

0.001 

0.019 

0.075 

0.174 

0.314 

0.483 

0.663 

0.827 

0.946 

0.997 

0.750 

0.000 

0.006 

0.027 

0.075 

0.158 

0.285 

0.456 

0.663 

0.866 

0.990 

0.850 

0.000 

0.001 

0.006 

0.019 

0.048 

0.105 

0.210 

0.389 

0.663 

0.956 

0.950 

0.000 

0.000 

0.000 

0.001 

0.003 

0.007 

0.020 

0.058 

0.182 

0.663 

For  binomial  parameters  of 

the  category  of  Interest 

;  near 

those 

Indicated  by 

the  data,  q-val 

ues  may  be  estimated  from  the 

following  table: 

9  \  * 

0.700 

0.72Q 

0.740 

0.760 

0.780 

0.800 

0.820 

0.840 

0.860 

0.880 

0.900 

0.805 

0.418 

0.460 

0.504 

0.551 

0.600 

0.650 

0.702 

0.753 

0.803 

0.852 

0.896 

0.825 

0.362 

0.402 

0.446 

0.493 

0.543 

0.595 

0.649 

0.705 

0.761 

0.816 

0.869 

0.845 

0.304 

0.342 

0.384 

0.429 

0.479 

0.532 

0.589 

0.648 

0.710 

0.772 

0.833 

0.865 

0.245 

0.280 

0.318 

0.361 

0.409 

0.461 

0.518 

0.580 

0.646 

0.716 

0.786 

0.885 

0.187 

0.217 

0.251 

0,289 

0.333 

0.382 

0.438 

0.500 

0.569 

0.644 

0.724 

0.905 

0.132 

0.156 

0.183 

0.215 

0.253 

0.297 

0.348 

0.407 

0.476 

0.553 

0.641 

0.925 

0.083 

0.100 

0.120 

0.144 

0.173 

0.208 

0.251 

0.302 

0.365 

0.440 

0.530 

0.945 

0.043 

0.052 

0.064 

0.079 

0.098 

0.122 

0.152 

0.190 

0.240 

0.304 

0.386 

0.965 

0.015 

0.018 

0.023 

0.030 

0.038 

0.049 

0.064 

0.085 

0.113 

0.153 

0.211 

0.985 

0.002 

0.002 

0.003 

0.004 

0.005 

0.007 

0.009 

0.013 

0.019 

0.028 

0.044 

Figure  3a.  First  half  of  printer  output  from  Figure  2  Input. 
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For  binomial  parameters  of  the  category  of  Interest  near  those  Indicated  by 
management,  q-values  may  be  estimated  from  the  following  table: 


9  \  4> 

0.700 

0.720 

0.740 

0.760 

0.780 

0.800 

0.820 

0.840 

0.860 

0.880 

0.900 

0.800 

0.431 

0.473 

0.518 

0.564 

0.613 

0.663 

0.713 

0.763 

0.812 

0.859 

0.902 

0.820 

0.375 

0.416 

0.460 

0.507 

0.557 

0.609 

0.663 

0.717 

0.772 

0.826 

0.876 

0.840 

0.318 

0.357 

0.399 

0.445 

0.495 

0.548 

0.604 

0.663 

0.723 

0.784 

0.842 

0.860 

0.259 

0.295 

0.334 

0.378 

0.426 

0.479 

0.536 

0.597 

0.663 

0.730 

0.798 

0.880 

0.201 

0.2,32 

0.267 

0.306 

0.351 

0.402 

0.458 

0.520 

0.589 

0.663 

0.740 

0.900 

0.145 

0.170 

0.199 

0.233 

0.272 

0.318 

0.370 

0.431 

0.499 

0.577 

0.663 

0.920 

0.094 

0.112 

0.134 

0.160 

0.192 

0.229 

0.274 

0.328 

0.393 

0.469 

0.559 

0.940 

0.051 

0.062 

0.076 

,0.093 

0.115 

0.141 

0.175 

0.217 

0.271 

0.338 

0.424 

0.960 

0.020 

0.025 

0.031 

0.040 

0.050 

0.064 

0.083 

0.108 

0.142 

0.189 

0.255 

0.980 

0.003 

0.004 

0.006 

0.007 

0.010 

0.013 

0.018 

0.025 

0.035 

0.052 

0.078 

1.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

For  binomial  parameters  of 

the  category  of  Interest 

i  near 

those 

indicated  by 

management,  q-values 

may  be  estimated  from  the  foil 

owing 

table! 

t 

6  \  * 

0.750 

0.770 

0.790 

0.810 

0.830 

0.850 

0.870 

0.890 

0.910 

0.930 

0.950 

0.770 

0.617 

0.663 

0.708 

0.753 

0.797 

0.839 

0.879 

0.914 

0.945 

0.969 

0.987 

0.790 

0.568 

0.615 

0.663, 

0.711 

0.759 

0.807 

0.852 

0.893 

0.930 

0.960 

0.982 

0.810 

0.513 

0.561 

0.611 

0.663 

0.715 

0.767 

0.819 

0.867 

0.911 

0.948 

0.976 

0.830 

0.453 

0.501 

0.553 

0.606 

0.663 

0.720 

0.777 

0.834 

0.886 

0.932 

0.968 

0.850 

0.389 

0.436 

0.487 

0.542 

0.601 

0.663 

0.726 

,0.790 

0.853 

0.909 

0.956 

0.870 

0.321 

0.365 

0.415 

0.469 

0.529 

0.593 

0.663 

0.735 

0.808 

0.877 

0.937 

0.890 

0.251 

0.290 

0.336 

0.387 

0.445 

0.511 

0.583 

0.663 

0.746 

0.831 

0.909 

0.910 

0.181 

0.213 

0.252 

0.297 

0.351 

0.413 

0.486 

0.569 

0.663 

0.764 

0.865 

0.930 

0.114 

0.138 

0.168 

0.204 

0.248 

0.302 

0.368 

0.449 

0.547 

0.663 

0.791 

0.950 

0.058 

0.072 

0.090 

0.113 

0.143 

0.182 

0.233 

0.301 

0.391 

0.510 

0.663 

0.970 

0.018 

0.023 

0.030 

0.040 

0.053 

0.071 

0.098 

0.137 

0.196 

0.288 

0.434 

For  binomial  parameters  of 

the  category  of  1i 

iterest  near 

those 

indicated  by 

management,  q-' 

val  ues 

may  be  estimated 

from  the  foT 

1  owing 

table 

• 

• 

0  N  <j> 

0.600 

0.620 

0.640 

0.660 

0.680 

0.700 

0.720 

0.740 

0.760 

0.780 

0.800 

0.900 

0.064 

0.076 

0.090 

0.105 

0.124 

0.145 

0.170 

0.199 

0.233 

0.272 

0.318 

0.920 

0.039 

0.046 

0.055 

0.066 

0.079 

0.094 

0.112 

0.134 

0.160 

0.192 

0.229 

0.940 

0.019 

0.023 

0.028 

0.035 

0.042 

0.051 

0.062 

0.076 

0.093 

0.115 

0.141 

0.960 

0.007 

0.008 

0.010 

0.013 

0.016 

0.020 

0.025 

0.031 

0.040 

0.050 

0.064 

0.980 

0.001 

0.001 

0.002 

0.002 

0.003 

0.003 

0.004 

0.006 

0.007 

0.010 

0.013 

1.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

Figure  3b.  Second  half  of  printer  output  from  Figure  2  Input. 
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ENTER  RELIABILITY-ENGINEER/THEORETICAL-STATISTICIAN  APPROACH  "R/S"  r 

ENTER  "TEST  ONE  NUMBER  OF  SUCCESSES"  20 
ENTER  "TEST  ONE  NUMBER  OF  FAILURES"  1 
ENTER  "TEST  TWO  NUMBER  OF  SUCCESSES"  11 
ENTER  "TEST  TWO  NUMBER  OF  FAILURES"  4 

ENTER  "T"  FOR  TABLE  OF  Q-VALUES,  "ANYTHING  ELSE"  TO  SKIP  TABLE  skip 

ENTER  ",C"  FOR  CLOSE  LOOK  AT  Q-VALUE  TABLE  IN  DATA  SUGGESTED  REGION, 
"ANYTHING  ELSE"  TO  SKIP  skip 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  e  ■  20  /  21  ■  0.952)  AND  THE 
SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  ■  11  /  15-0.733), 
"ANYTHING  ELSE"  TO  SKIP  m 

ENTER  V  .9 

ENTER  V  .8 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  e  -  20  /  21  ■  0.952)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  -  11  /  15  -  0.733), 

"ANYTHING  ELSE"  TO  SKIP  ffl 

ENTER  "e"  1 

ENTER  V  -7 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  9  ■  20  /  21  ■  0.952)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  a  •  11  /  15  •  0.733), 

"ANYTHING  ELSE  TO  SKIP  skip 

END  OF  PROGRAM 


Figure  4.  Sample  of  reliability-engineer  Input. 
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ANALYSIS  OF  FISHER'S  EXACT  TEST 

Although  Test  One,  Test  Two,  Successes,  and  Failures  may  be  Interchanged 
several  ways  mathematically,  they  have  physical  Identities,  To  utilize  these 
Identities,  [A]  Test  One  (i.e.  the  test  with  20  Successes  and  1  Failure)  Is 
taken  as  the  sample  of  prominence  (I.e.  It  Is  considered  physical ly  more 
important  than  Test  Two)  and  [B]  Successes  define  the  category  of  interest 
(i.e.  the  most  natural  description  of  a  test  result  is  considered  to  be 
Success  instead  of  Failure). 

In  the  following  standardly  ordered  table,  the  sample  of  prominence  and 
category  of  Interest  are  Identified  by  the  user  as  Sample  I  and  Category  1. 

Category  1:  Category  2: 

Sample  I:  r  ■  20  n  -  r  ■  1  n  ■  21 

Sample  II:  R  -  r  ■  11  (N  -n)  -  (R  -  r)  ■  4  N  -  n  ■  15 

R  -  31  N  -  R  ■  5  N  ■  36 

For  this  data,  the  post-test  risk  of  a  Type  I  error  is  p-value  ■  0.084. 

For  this  data's  two  suggested  binomial  parameters  of  the  category  of  Interest 
(I.e.  e  ■  20  /  21  ■  0.952  and  *  ■  11  /  15  *  0.733),  the  post-test  risk  of 
a  Type  II  error  Is  q-value  «  0.241. 


For  binomial  parameters  of  the  category  of  Interest  near  those  Indicated  by 
management,  q-values  may  be  estimated  from  the  following  table: 


9  \  4 

0.700 

0.720 

0.740 

0.760 

0,780 

0. 800 

0.820 

0.840 

0.860 

0,880 

0.900 

0.800 

0.785 

0.815 

0.843 

0.869 

0.894 

0.916 

0.936 

0.953 

0.968 

0.979 

0.988 

0.820 

0.741 

0.774 

0.806 

0.837 

0.865 

0.892 

0.916 

0.937 

0.956 

0.971 

0.983 

0,840 

0.690 

0.726 

0.761 

0.795 

0.829 

0.860 

0.889 

0.916 

0.939 

0.959 

0.975 

0.860 

0,628 

0.667 

0.705 

0.744 

0.782 

0.818 

0.853 

0.886 

0.916 

0.942 

0.964 

0.880 

0.556 

0.596 

0.637 

0.679 

0.721 

0.763 

0.804 

0.844 

0.882 

0.916 

0.945 

0.900 

0.471 

0.511 

0.553 

0.597 

0.643 

0.690 

0.737 

0.785 

0.832 

0.876 

0.916 

0.920 

0.374 

0.411 

0.452 

0.496 

0.543 

0.592 

0.645 

0.700 

0.756 

0.812 

0.867 

0.940 

0.265 

0.297 

0.333 

0.372 

0.416 

0.465 

0.518 

0,577 

0.641 

0.709 

0.780 

0.960 

0.152 

0.174 

0.199 

0.229 

0.263 

0.302 

0.349 

0.403 

0.466 

0.539 

0.623 

0.980 

0.050 

0.059 

0.070 

0.083 

0.099 

0.118 

0.143 

0.174 

0.214 

0.267 

0.337 

1.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

For  binomial  parameters  of  the  category  of  Interest  near  those  Indicated  by 


management,  q- 

•values  may  be  estimated 

from  i 

the  fol 

lowing 

table: 

0  \  * 

0.600 

0.620  0.640 

0.660 

0.680 

0.700 

0.720 

0.740 

0.760 

0.780 

0.800 

0.900 

0.302 

0.332  0.363 

0.397 

0.433 

0.471 

0. 511 

0.553 

0.597 

0.643 

0.690 

0.920 

0.225 

0.250  0.277 

0.307 

0.339 

0.374 

0.411 

0.452 

0.496 

0.54  3 

0.592 

0.940 

0.149 

0.168  0.188 

0.211 

0.237 

0,265 

0.297 

0.333 

0.372 

0.416 

0.465 

0.960 

0.079 

0.090  0.102 

0.117 

0.133 

0.152 

0.174 

0.199 

0.229 

0.263 

0.302 

0.980 

0.024 

0.027  0.032 

0.037 

0.043 

0.050 

0.059 

0.070 

0.083 

0.099 

0.118 

1.000 

0.000 

0.000  0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

Figure 

5.  Printer  output 

from  Figure 

4  input. 
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ENTER  RELIABILITY-ENGINEER/THEORETICAL-STATISTICIAN  APPROACH  "R/S"  S 

ENTER  SIZE  OF  POPULATION  "N"  36 

ENTER  SIZE  OF  SAMPLE  OF  PROMINENCE  "n"  21 

ENTER  #  OF  ITEMS  FROM  CATEGORY  OF  INTEREST  IN  POPULATION  "R"  31 

ENTER  #  OF  ITEMS  FROM  CATEGORY  OF  INTEREST  IN  SAMPLE  OF  PROMINENCE  "r"  18 

ANALYSIS  OF  FISHER'S  EXACT  TEST 

In  the  following  standardly  ordered  table,  the  sample  of  prominence  and 
category  of  Interest  are  Identified  by  the  user  as  Sample  I  and  Category  2. 

Category  1:  Category  2: 

Sample  I:  r  *  3  n  -  r  ■  13  n  -  21 

Sample  I 1 :  R  -  r  ■  2  (N  -n)  -  (R  -  r)  ■  13  N  -  n  ■  15 

R  ■  5  N  -  R  ■  31  N  -  36 

For  this  data,  the  post-test  risk  of  a  Type  I  error  is  p-value  ■  0.663. 

For  this  data's  two  suggested  binomial  parameters  of  the  category  of  Interest 
(l.e.  o  *  18  /  21  ■  0.857  and  *  ■  13  /  15  *  0.867),  the  post-test  risk 
of  a  Type  II  error  Is  q-value  ■  0.306. 

ENTER  "T"  FOR  TABLE  OF  Q-VALUES,  "ANYTHING  ELSE"  TO  SKIP  TABLE  skip 

ENTER  "C"  FOR  CLOSE  LOOK  AT  Q-VALUE  TABLE  IN  DATA  SUGGESTED  REGION, 

"ANYTHING  ELSE"  TO  SKIP  skip 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  o  ■  18  /  21  ■  0.857)  AND  THE 
SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  *  -  13  /  15  ■  0.867), 

"ANYTHING  ELSE"  TO  SKIP  m 

ENTER  "0"  .8 

ENTER  V  .9 

For  binomial  parameters  of  the  category  of  interest  near  those  indicated  by 
management,  q-values  may  be  estimated  from  the  following  table: 

8  \  *  0.700  0.720  0.740  0.760  0.780  0.800  0.820  0.840  0.860  0.880  0.900 

0.800  0.158  0.184  0.215  0.250  0.291  0.337  0.391  0.452  0.521  0.598  0.682 

0.820  0.127  0.149  0.176  0.207  0.244  0.287  0.337  0.396  0.464  0.542  0.630 

0.840  0.098  0.117  0.139  0.166  0,198  C.237  0.283  0.337  0.403  0.480  0.569 

0.860  0.073  0.088  0.106  0.128  0.155  0.188  0.228  0.277  0.337  0.411  0.501 

0.880  0.051  0.062  0.076  0.093  0.114  0,141  0.174  0.216  0.270  0.337  0.423 

0.900  0.033  0.041  0.050  0.063  0.078  0.098  0.124  0.158  0.202  0.260  0.337 

0.920  0.019  0.024  0.030  0.037  0.048  0.061  0.079  0.103  0.136  0.182  0.246 

0.940  0.009  0.011  0.014  0.019  0.024  0.032  0.042  0.057  0.077  0.108  0.154 

0.960  0.003  0.004  0.005  0.007  0.009  0.012  0.016  0.022  0.032  0.047  0.071 

0.980  0.000  0.001  0.001  0.001  0.001  0.002  0.003  0.004  0,006  0.009  0.015 

1.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000 

Figure  6,  Input  and  screen  output  in  case  that  alters  order  In  table. 
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ENTER  RELIABILITY-ENGINEER/ THEORETICAL -STATISTIC IAN  APPROACH  "R/S"  r 

ENTER  "TEST  ONE  NUMBER  OF  SUCCESSES"  11 
ENTER  "TEST  ONE  NUMBER  OF  FAILURES"  3 
ENTER  "TEST  TWO  NUMBER  OF  SUCCESSES"  107 
ENTER  "TEST  TWO  NUMBER  OF  FAILURES"  14 

ANALYSIS  OF  FISHER'S  EXACT  TEST 

Although  Test  One,  Test  Two,  Successes,  and  Failures  may  be  Interchanged 
several  ways  mathematically,  they  have  physical  Identities.  To  utilize  these 
Identities,  [A]  Test  One  (l.e.  the  test  with  11  Successes  and  3  Failures) 
Is  taken  as  the  sample  of  promlnlnce  (l.e.  It  Is  considered  physical ly  more 
Important  than  Test  Two)  and  [B]  Successes  define  the  category  of  Interest 
(l.e.  the  most  natural  description  of  a  test  result  Is  considered  to  be 
Success  Instead  of  Failure). 

In  the  following  standardly  ordered  table,  the  sample  of  prominence  and 
category  of  Interest  are  Identified  by  the  user  as  Sample  II  and  Category  1. 

Category  1:  Category  2: 

Sample  I:  r  ■  107  n  -  r  ■  14  n  ■  121 

Sample  II:  R  -  r  ■  11  (N  -n)  -  (R  •  r)  *  3  N  -  n  ■  14 

R  ■  118  N  -  R  -  17  N  -  135 

For  this  data,  the  post-test  risk  of  a  Type  I  error  Is  p-value  ■  0.249. 

For  this  data's  two  suggested  binomial  parameters  of  the  category  of  Interest 
(l.e.  4  ■  11  /  14  -  0.786  and  e  ■  107  /  121  ■  0.884),  the  post-test  risk  of 
a  Type  II  error  Is  q-value  ■  0.375. 

ENTER  "T"  FOR  TABLE  OF  Q-VALUES,  "ANYTHING  ELSE"  TO  SKIP  TABLE  t 

For  other  binomial  parameters  of  the  category  of  Interest,  q-values  may  be 
estimated  from  the  following  table: 

9\  0.050  0.150  0.250  0.350  0.450  0.550  0.650  0.750  0.850  0.950 

0.C50  0.751  0.981  0.996  0.999  1.000  1.000  1.000  1.000  1.000  1.000 

0.150  0.154  0.751  0.926  0.976  0.992  0.997  0.999  1.000  1.000  1.000 

0.250  0.022  0.429  0.751  0.897  0.959  0.985  0.995  0.999  1.000  1.000 

0.350  0.003  0.190  0.516  0.751  0.884  0.951  0.982  0.995  0.999  1.000 

0.450  0.000  0.064  0.289  0.549  0.751  0.880  0.951  0.985  0.997  1.000 

0.550  0.000  0.016  0.124  0.329  0.559  0.751  0.884  0.959  0.992  1.000 

0.650  0.000  0.002  0.035  0.144  0,329  0.549  0.751  0.897  0.976  0.999 

0.750  0.000  0.000  0.005  0.035  0.124  0.289  0.516  0.751  0.926  0.996 

0.850  0.000  0.000  0.000  0.002  0.016  0.064  0.190  0.429  0.751  0.981 

0.950  0.000  0.000  0.000  0.000  0.000  0.000  0.003  0.022  0.154  0.751 

Figure  7a.  First  half  of  rel iabll ity-englneer  Input  and  screen  output 
for  hypothetical  missile  modification  analysis. 
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ENTER  "C"  FOR  CLOSE  LOOK  AT  Q-VALUETABLE  IN  DATA  SUGGESTED  REGION, 

"ANYTHING  ELSE"  TO  SKIP  c 

For  binomial  parameters  of  the  category  of  Interest  near  those  Indicated  by 
the  data,  q-values  may  be  estimated  from  the  following  table: 

8\  4  0.686  0.706  0.726  0.746  0.766  0.786  0.806  0.826  0.846  0.866  0.886 

0.784  0.499  0.550  0.603  0.655  0.706  0.755  0.801  0.844  0.882  0.916  0.943 

0.804  0.430  0.483  0.537  0.593  0.648  0.702  0.755  0.805  0.850  0.891  0.926 

0.824  0.358  0.409  0.464  0.521  0.580  0.539  0.698  0.755  0.809  0.859  0.902 

0.844  0.283  0.331  0.384  0.441  0.501  0.564  0.628  0.693  0.756  0.815  0.869 

0.864  0.208  0.251  0.299  0.353  0.412  0.476  0.544  0.614  0.686  0.756  0.823 

0.884  0.138  0.173  0.213  0.260  0.314  0.375  0.443  0.517  0.595  0.676  0.757 

0.904  0.079  0.103  0.132  0.168  0.212  0.265  0.327  0.399  0.480  0.569  0.663 

0.924  0.035  0.048  0.065  0.088  0.118  0.156  0.205  0.265  0.339  0.427  0.530 

0.944  0.010  0.014  0.021  0.031  0.045  0.065  0.093  0.131  0.184  0.255  0.349 

0.964  0.001  0.002  0.003  0.005  0.007  0.012  0.020  0.033  0.053  0.087  0.140 

0.984  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.001  0.002  0.003  0.008 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  *  ■  11  /  14  -  0.786)  AND  THE 

SAMPLE  OF  NON-PROMINENCE  (I.E.  REPLACE  e  -  107  /  121  ■  0.884), 

"ANYTHING  ELSE"  TO  SKIP  m 

ENTER  V  .85 

ENTER  V  .9 

For  binomial  parameters  of  the  category  of  Interest  near  those  Indicated  by 
management,  q-values  may  be  estimated  from  the  following  table: 

9  \  *  0.750  0.770  0.790  0  810  0.830  0.850  0.870  0.890  0.910  0.930  0.950 

0.800  0.618  0.673  0.726  0.776  0.824  0.867  0.905  0.937  0.962  0.980  0.992 

0.820  0.550  0.608  0.666  0.724  0.778  0.830  0,876  0.916  0.948  0.973  0.989 

0.840  0.471  0.532  0.595  0.658  0.721  0.781  0.837  0.887  0.929  0.962  0.984 

0.860  0.384  0.445  0.510  0.578  0.648  0.717  0.785  0.847  0.901  0.945  0.976 

0.880  0.291  0.348  0.411  0.481  0.556  0.634  0.712  0.789  0.860  0.919  0.964 

0.900  0.197  0.245  0.302  0.368  0.442  0.525  0.614  0.706  0.7%  0.877  0.942 

0.920  0.110  0.145  0.189  0.243  0.310  0.389  0.482  0.585  0.696  0.805  0.902 

0.940  0.044  0.063  0.088  0.123  0.170  0.232  0.313  0.415  0.539  0.679  0.822 

0.960  0.009  0.014  0.022  0.034  0.054  0.084  0.131  0.201  0.306  0.454  0.646 

0.980  0.000  0.000  0.001  0.002  0.003  0.006  0.013  0.026  0.056  0.122  0.265 

1.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000  0.000 

ENTER  "M"  FOR  MANAGEMENT  INDICATED  BINOMIAL  PARAMETERS  TO  REPLACE 
THE  DATA  INDICATED  PARAMETERS  OF  THE  CATEGORY  OF  INTEREST  IN  THE 
SAMPLE  OF  PROMINENCE  (I.E.  REPLACE  *  -  11  /  14  -  0.786)  AND  THE 

SAMPLE  OF  NON-PROMENENCE  (I.E.  REPLACE  0  -  107  /  121  -  0.884), 

"ANYTHING  ELSE"  TO  SKIP  skip 

END  OF  PROGRAM 

Figure  7b.  Second  half  of  hypothetical  missile  modification  analysis. 
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